« 3.013: of public and private clouds - in 98 seconds | Main | 3.015: mmphmy hlmmphums, emphmgmy! »

October 08, 2010

3.014: so much 'ado about . . .

Try to keep up, now! Wow – what a bustling couple of weeks!

So many competitive storage announcements, you'd practically think they were all scheduled to maximize their disruptive impact on Q4 storage spend.

When you're the market leader, as EMC has been for the past 2 decades-plus, you learn to expect this almost annual frenzy. It comes with the target that leaders have tattooed on their backs.

This year the wanna-bee followers seem particularly agitated, though. Hitachi invested heavily in marketing sizzle for the first time since Mr. T was their chosen spokesperson – and with good reason, I'll admit: by my observations of IDC Storage Tracker data, Hitachi's delay in refreshing the aged USP-V (coupled with the loss of Sun as a reseller) has driven 5 straight quarters of USP-V market share declines vs. VMAX and the newly retired IBM DS8700.

Hitachi obviously had to try something different, even if it meant moving to yet-another new processor base. But unable to change their architecture to fully leverage industry-standard open components, their "rush" to market was slowed by the need to create FOUR proprietary ASICs. And those ASICs further handcuffed the move to the Intel platform. With the unavoidably long lead-times of ASIC development, Hitachi was locked into implementing with the PCIe Gen1-based infrastructure and processors, even as Intel is delivering the second-generation of PCIe Gen2 CPUs and interconnect. The net result? Using the same Intel processor as the 19-month old VMAX, the new VSP can't even double the performance of the USP-V that it doesn't quite replace.

That leaves VMAX at the top of the performance heap, having more than doubled the performance of the DMX4 when it was introduced in April 2009.

As a further testament to the insignificance of the VSP, I'll also note that HP has chosen to use a totally different name for the product in their lineup. Not only has my old boss shunned the brand, he and his new head of storage outright told the world that Hitachi remained in the product lineup only to support HP's mainframe customers and to fill the void above 3PAR until such time as it grows up. That must have thrilled HP EVA and XP customers alike, both groups who now find themselves sitting on dead-end kit with no defined escape path.

But that's a story for another day…

Yep, beneath all the new-found bravado and marketing spin, the simple reality is that Hitachi & HDS are still attempting to follow EMC's lead, and they're falling further and further behind.

Over at IBM, someone over there is apparently tired of getting left behind in the storage space. That someone must have an MBA as well, since they've apparently figured out that bundling storage with servers at no extra charge is not a long-term profitable strategy – especially when so much of the server base is shifting to Intel-based compute clusters rather than proprietary hardware. Hard times lead to big gambles, and the industry analysts apparently think that IBM has gone all-in on this one.

Like Hitachi, IBM has sort of replaced its high-end DS8700 with the new DS8800. I say "sort of" because the DS8800 doesn't yet support all the same features that the DS8700 does, so IBM is keeping the old array on life support for those that need esoteric stuff like:

Other limitations
The following functions are currently not available on DS8800:

  • Quick initialization and thin provisioning support
  • Remote Pair FlashCopy support
  • Easy Tier support
  • Multiple Global Mirror session support
  • z/HPF extended distance capability support
  • z/OS distributed data backup support
  • IBM Disk full page protection support
  • 16 TB LUN size is not available

(from the IBM DS8800 announcement materials)

IBM made lots of noise about how the DS8800 reused 95% of the DS8700 code, but I'm skeptical of that claim – I find it hard to believe that thin provisioning, Easy Tiering, Global Mirror and the rest represent 5% of unique code. TonyP asserted that the above features were "forked out" of the main development tree to be reunited sometime in 1H'2011 – I read that as admitting the DS8800 was rushed to hit a date for marketing value rather than customer value. With all their hype around "green,"

I find it particularly interesting that thin provisioning and Easy Tier were deferred by the powers at IBM, since the very purpose of both is to improve utilization (reduce wasted/unused capacity), reduce footprint and lower power and cooling requirements (fewer drives), and to improve overall performance and efficiency (thanks to more IOPS/drive with SSDs and more GB/drive with SATA). Seems like the IBM sales teams are going to have to speak out of both sides of their mouths until the rest of the DS8800 software is designed, developed and debugged.

Finally, I haven't forgotten the newly-minted V7000 array that pretty much my took pal @BarryW off-line since the beginning of 2009. First – hats off to you sir…you've been lobbying to turn that software into a storage array for far too long. And thankfully, you appear to have overcome the insistence of the publicly-anointed father of storage, relegating his XIV abomination to merely a footnote in the announcement (in fact, the only reference to XIV was to call out that the new SVC GUI was modeled after the pooled simplicity of XIV's UI).

But like the DS8800, the new V7000 clearly isn't quite ready for the market either…sure, the code base is stable – it's been out there quite some time now. In fact, you might say it's beginning to show its age, being as it still lacks the true scale-out clustering and distributed active/active cache coherency that fuels EMC's VPLEX. I suspect the newly formed V-team found it a bit more challenging to handle all the enclosure and spindle management that was originally envisioned for the product – the result being a dual-controller mid-tier (barely) array that can support only a tiny fraction of the capacity supported by EMC's Unified platform. It's a start, I guess, but it clearly isn't going to be the CLARiiON-killer IBM was boasting about yesterday – at least not until it scales up to a real processor and something better than 240 drives (the max drives the V7000 supports out the gate is only 120 according to the announcement materials)

I gather from the blatant exclusion of XIV from IBM's obviously coordinated and planned Major Storage Announcement Event that someone has finally figured out that the XIV architecture's inherent fallibility was too much of a liability. Given the rate that VMAX and CLARiiON are replacing XIV arrays that haven't lived up to the hype, coupled with the snub at this week's event, I figure Moshe left IBM for much the same reason Randy Moss left the Patriots.

Nobody wants to work where they aren't appreciated, right?

That's all for now, but I'll undoubtedly have more to say about all this "ado" in the coming days and weeks.

TTFN! 
 


TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d834c659f269e20134880e95d2970c

Listed below are links to weblogs that reference 3.014: so much 'ado about . . .:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Barry Whyte

Well there's a shocker - you come out of hiding to post about HDS and IBM again - nothing about EMC... apart from the obviously aged and maybe its dead CX platform? Got problems there? Where's the SAS drives, wheres the 2.5" SFF drives? 4 Gbit FC-AL how quaint !!!!

Ooops, hang on a minute, looks like I switched surname for a second!

No issues with backend drive here thanks, just reduce test load at initial GA to ensure day1 config is 100% stable.

And I think most people will find that the 240 drive system will happily out-perform a CX-4 even 960... how many actually have 960 drives, and how many of those drives can you actually saturate - even half of them? ??

Dctools.blogspot.com

I can agree with many of your comments until getting down to the V7000/SVC comparison.

I don't really think it's fair to compare the SVC to the VMAX, they're in a different class of competition. Now compare it to the VPLEX, where it has a hard time scaling beyond 2 nodes in geographically dispersed areas, or it's lack of write cache. (And Invista is dead now, right?)

The V7000, while incorporating SVC-like features may take it farther. Time will tell. It least auto tiering is included free of charge.

Not that I don't think EMC has some real answers and leadership. Tiering beyond 2 tiers is nice, thanks EMC.

I think the earlier opinions prior to SVC/V7000 may have hurt, but been better placed, but I'd rather see you hype up your strengths. You do have some nice ones.

Joshkrischer

What a collection of FUDs and inaccuracies. It would take me too much time to refute most of your statements or assumption.
Talking about aged subsystems how many years EMC “stretched” the original Symmetrix or the DMX? What are the differences between DMX-3 and DMX-4?
I told you once already that “ People Who Live In Glass Houses Should Not Throw Stones”.
BTW, is FAST II supported on state-of-the-art VMAX?

the storage anarchist

Urban -

Thanks for the comments, and welcome to the blog-o-sphere.

First, I couldn't agree with you more: an array built upon SVC's two-node cluster architecture is clearly not to be compared with enterprise-class, multi-controller arrays.

Your assertion of VPLEX's scalability is curious, as a single VPLEX cluster can scale from 2 to 8 nodes (unlike SVC which can only add 2-node clusters in a loose federation). With VPLEX, each additional node adds to the global memory, increases front-end and back-end connectivity, and scales performance linearly. Every VPLEX node can service I/O requests from any host, and can deliver I/O to any connected storage array - all with total cache coherency.

What VPLEX can do that NOTHING else can do is to tightly Federate two separate VPLEX clusters, each of which can have from 2-8 nodes. This "federation" is called "AccessAnywhere" and it allows the two clusters to "stretch" one or more LUNs across the two clusters. These LUNs are presented to hosts connected to either (or both) clusters fully active/active and with total cache coherency. Reads at either cluster will always access the latest written data, no matter where written.

The closest SVC (or NetApp) can come to this VPLEX functionality is to split their two-node cluster into two physically separate halves. In this configuration, neither of the halves is HA, and neither have any protection against failures other than their (distant) other half. With VPLEX, both clusters remain fully HA and can tolerate virtually any failure without interruption of I/O or loss of protected status (again, since every node in a VPLEX cluster can service any I/O).

I'll be talking about more differences in future posts - this was just a quick one to put a stick in the ground around which we can have some intersting conversations.

Finally, indeed EMC has many advantages and strengths. That's why we have market share leadership, and why we have a target tatooed on our backs (as the next commenter demonstrates).

Thanks!

the storage anarchist

Josh -

It is somewhat amazing that you continue to attack Symmetrix even though you've not taken the time to truly understand how its architecture has evolved over the past 21 years. As its continued leadership in market share demonstrates, good designs are timeless.

But you are right on one point: the differences between the hardware of DMX-3 and DMX-4 was primarily the change to the switched loop back end and the support for 3.5" SATA drives (more than a year before either IBM or Hitachi supported SATA in their so-called enterprise arrays).

What was MORE IMPORTANT about the move from DMX3 to DMX4, however, was the fact that the SAME MICROCODE SUPPORTED BOTH. Thus, all the new features introduced with DMX4 - thin provisioning, VLUN, RAID 6, QoS, etc. - all these features were also supported on DMX3 systems. Contrast that to IBM who merely upgrade the processor and change the back-end to create the DS8800, yet they REMOVE functionality that is available on the DS8700.

As to sub-LUN FAST on VMAX - it's coming. Unlike both IBM and Hitachi who have rushed to market their wanna-be competing alternatives, we're taking the time to do it right. As you'll soon see, moving sub-LUN extents of data at 1GB (IBM) or 42MB (Hitachi) once a day is a very, very inefficient granularity for automated tiering. A robust solution must be simple to set up and manage, dynamic and adaptable to change. Supporting more than 2 tiers is also a prerequisite to optimal price/performance efficiency.

The deficiencies in implementation are understandable, however, since neither company is apparently willing or able to make the architectural changes necessary to their systems to keep up with the dynamics of the virtual data center.

Oh, and speaking of "where's that feature" - what ever happened with HHAM? More than a year after announcement, and still absolutely zero customer sightings. That little delay is allowing VPLEX to sweep the floor around the globe, owing that it is the only active/active clustering solution available!

Glass Houses, indeed!

Joshkrischer

Barry, my question was rhetoric pointing to your “aging “ remarks. I understand the 3 different hardware designs pretty well. I spent enough time with Moshe Yanai to understand the bus based Sym 1-Sym 5. The DMX is based on the Cereva Networks Inc design and the loosely coupled clustered design of the Vmax. I will present on Storage control units structures on the SNW EMEA.
The size of the “chunklets” in sub-LUN design have very strong influence on internal data movements overhead and the subsystem performance.
There is a big difference between the ability to deliver a function and the ability to sell it therefore your remark on the HHAM is irrelevant.
BTW, are sure that the Vplex is “sweeping the floor around the globe” ? The Vplex installation in one of my customer’s site was delayed due to a microprogram errors.

the storage anarchist

Josh -

Although I personally have explained the facts behind the DMX architecture to you on several occasions, you insist on creating your own version of history.

Fact: the engineers who designed the DMX architecture did not have access to the Cereva IP. They couldn't have - Cereva's assets were acquired by EMC *after* the DMX was in implementation. The VMAX architects likewise did not have access (nor did they need access) to the Cereva IP...both innovations were entirely developed in-house, contrary to what you think or say.

You can continue to spin your tall tales, but you do yourself a disservice by doing so.

As to auto-tiering, indeed, the size of the extents has an impact, and the optimal solutions combine both innovations in hardware and software. But no matter how you spin it, the more data you have to move as a unit increases both the time and overhead required to promote & demote across the tiers. Bigger is thus not always better, nor is delaying optimization for 24 hours (or longer).

I have no idea what your comment on HHAM is supposed to mean: HHAM was announced last year as generally available, yet there have been no evidence of it actually shipping to customers for EITHER the HA clustering of USP-V's or the non-disruptive migration use cases.

Finally, I will not directly respond to your unverifiable attempt to create FUD around VPLEX. Like I said to your old pals at IBM, such competitive tactics are not unexpected given the accellerating successes of both VMAX and VPLEX.

Joshkrischer

Barry,
1) Moshe Yanai briefed me on Symmetrix road map and matrix based DMX was not there. It is hard to believe that EMC needed less than two years to develop a complete new subsystem (in particular after significant developer exodus in 2001 and 2002) moving from bus-based to switch based. Porting Symmetrix software on new hardware is more logical.
2) In June 2002 EMC acquired storage startup Cereva Networks, Inc. and in February 2003 EMC announced the DMX, what a coincidence.
3) Not using Cereva design? Not letting developers to mix? Then why EMC acquired Cereva? EMC could hire the 40 developers with minimal costs. I always considered EMC as a ruthless business machine with a hawk eyes and speed for acquisitions but never as a Samaritan organization that invest 10 Millions in a falling company without any use.
4) In 2003 when an ex-Cereva developer accidently seen my Gartner slides on DMX his reaction was “ it is our Cereva!”
5) I don’t spread FUD. A customer asked me if I heard about other users of Vplex with firmware 4.1 not supporting recovery for all cases of errors for VMware HA Cluster. Please remember the “ People Who Live In Glass Houses Should Not Throw Stones”- nobody in the industry is spreading more FUD than your blog.

the storage anarchist

Josh -

The fact Moshe didn't show you DMX on the roadmap doesn't mean it did not exist at the time. Moshe was often selective with what he shared with whom.

Your own timeline proves that DMX was not based on Cereva design. In June 2002, the hardware implementation was complete and FPGAs cast in hardcopy already. There is no way that IP acquired in June 2002 could have been incorporated into DMX that late in the development cycle.

I cannot comment on the reasons for any of EMC's acquisitions, but as the marketing manager responsible for launching DMX, i can assure you that your version of history is built upon coincidence, circumstance and your imagination - but not the facts.

VPLEX support for VMware HA covers a broad range of use cases and disruptions, more than any other multi-site offering in fact. That said, a complete HA (or FT) solution requires additional integration between VPLEX Metro and VMware, work we can discuss with customers under NDA.

If you ever decide not to be100% anti-EMC any longer, let me know. I'd be happy to welcome you back and share with you more details about our current and future products.

Thanks!

the storage anarchist

Josh -

Here is the core patent for the Symmetrix Direct Matrix Architecture.

As you can see, it was filed on Dec 30, 1998 and granted on May 14, 2002.

Contrary to the misinformation about DMX that you have been delivering for so long, any similarity to any other product is purely coincidental.

And you'll have to ask Moshe why he didn't tell you about DMX...I guess it is possible that he didn't know about it before he left EMC.

Roberto

I had started a long response when someone stepped in my office and made lose all the work done. Well, may be I will be a little less angered in this mine now.

What a provocation and misinformation; stuff the old USSR politburo would be proud of.

HP using a different name, oh my oh my....never noticed that it was such since...1998 when they dumped EMC and resold HItachi? or is it EMC that dumped them? I can still find the papers of that time to check.

HDS lost Sun channel; true and it was in the air for quite sometime; as a matter of fact, the vultures were around them to get the business w/o realizing that they were actually going out of business. HDS was ready and we covered the whole revenue we used to get from SUn and more in our fiscal 2009. BTW, while we are tal;king about 2009, we were actually doing well with our poor USP V line, growing revenue beyond expectations. True, lower expectations because, in case you have not noticed, there was a big recession going on. DMX business declined much more than USP V business and you don't need the IDC tracker to know that.

Talking about DMX...teh USP has been refreshed 3 years after its announcement; three years to this date....how long has been since teh D

Technology refresh: the VSP is not just the same old with a new processor, that's EMC approach. We took a proven technology and added a new component that expedite performance, functionality and enable new capabilities. In the meantime, we were redesigning it to make it consume a lot less, making it smaller to fit in a standard rack, added SAS/SFF while you are still on the old ones....

Yes, we didn't come out with SSD till 8 months after EMC first announcement and we proved that the market was not so hot fortehm. W/o intelligence to manage them and lower their relative cost, it was not something people would jump to have them. I think it is well known what happen to EMC at the beginning of the year....oversold the idea and overbought the product. Now the VSP has that capability and we start to see interest...by mixing them with Dynamic Tiering. In teh meantime, FAST2 has come out and iot is improved...you gave it a new name; typical...new coat of paint for something that it is at least a year behind schedule (spinning as you like).

I'll stop here now, I've real work to do....

the storage anarchist

Roberto -

Thank you for taking the time to comment.

Your assertions of market share are not supported by the facts of Storage Tracker, and i notice that recently Gartner has moved Hitachi into the "Other" category when comparing external storage suppliers.

But it's always fun to see how competitors spin the facts...don't you think so?

Josh Krischer

Barry,
This patent was initially issued in 1988 (and updated several times, the last update by Ofer Erez in 2000 close to the end of life of the original Sym) for the original bus-based Symmetrix, see the title:"System for interfacing a data storage system to a host utilizing a plurality of busses for carrying end-user data and a separate bus for carrying interface state data" and not for matrix based DMX.
Barry,
This patent was initially issued in 1988 (and updated several times, the last update by Ofer Erez in 2000 close to the end of life of the original Sym) for the original bus-based Symmetrix, see the title:"System for interfacing a data storage system to a host utilizing a plurality of busses for carrying end-user data and a separate bus for carrying interface state data" and not for matrix based DMX.
In 2001 the road map of EMC (according to Moshe) was adding more busses and increasing the bandwidth of the busses or you want to say that EMC Senior VP cheated Gartner analyst by providing wrong information.
Did EMC had an underground development division, hiding information from the head of Engineering? It is hard to believe!

the storage anarchist

Josh -

I know personally the engineers who wrote this patent and who architected DMX, and this is indeed the core patent behind it.

As to what Moshe told you, you'll have to ask him.

Your bias against EMC and your imagined reality serve only to undermine your credibility and is an insult to the good men and women who invented Symmetrix and DMX.

And with that, I'm done - I'm not going to play this game with you any more. If you want the last word, fire away.

Ken Steinhardt

Josh, I've been in and out of the thick of all of this for the last 16 years, and as Barry states, your assertions really are unfounded. I also certainly can't speak for whatever Moshe Yanai might have said to you, but I do know what we were building, and when, and the facts contradict your statements. Your agenda for presenting fantasy and ignoring hard facts appears to be pretty obvious, and I've personally seen you do it before in a conversation that I had with you as part of a larger group meeting back in 2002 that you were dialed in to when you refused to explore a suggestion that I had made to have discussions with some very specific customers whose hard experiences contradicted a position that you had taken in one of your notes. I see you taking a consistent pattern of stating strong positions (always against EMC) and refusing to hear or explore contradictory facts.

Joshkrischer

It is amazing that that every time that I point to inaccuracies in their blogs, EMC bloggers instead of delivering sound answers are attacking my credibility. It seems that EMC culture is not mature to keep discussions on technical facts without going personal.

Barry,
1. The patent which you mentioned is for the Bus structured Symmetrix not for DMX.
2. The meeting with M. Yanai was an official meeting on EMC roadmap organized by Steve Bardige, the head of EMC Analyst relations who also took part in the meeting.
3. You answer is marketing stuff without any “beef”
Ken,
1. What you both say is EMC developed the DMX without the knowledge of the head of engineering or you EMC developed the DMX in less than one year time. Who, with some technical understanding do you think will believe that?
2. In 2002 I pointed that the Symmetrix is aged design and its bandwidth is too small to support large capacity. EMC reaction was sending lawyer letter to Gartner management. Let me to quote from Storage Anarchist blog: July 03, 2008, 1.015: stranger danger: “From the outset, Symmetrix DMX has contradicted many of the design tenets that Moshe seems to embrace even still today. Some will even say that the Symm 5 (the 8000 series) embodiment of those design tenets was no longer competitive in 2001 or 2002, and that Symmetrix fell behind the competition and nearly cost EMC it's market-leading position”. Same conclusion but 6 years later.
3. EMC complained xx times to Gartner about my writing but was not even once to prove that it was wrong. I invite you to read my Research Notes looking for wrong information. The only two times which I was wrong was because I was deliberately misled by EMC; GDPS support by IT Austria and SRDF/A performance.
4. Never in my life have I refused to speak with any reference customer! I cannot recall any call that we had in the past. Please refresh my memory.
5. I am not biased against EMC; in fact one of my customers is deploying VMware in his main and DR data centers following my recommendation. I can give the name if you want, large storage RFP in Israel, etc.
6. Using the same logic it is unethical for any vendor employed bloggers to write about his competition, and in particular inaccuracies and FUD.
7. “I see you taking a consistent pattern of stating strong positions (always against EMC) and refusing to hear or explore contradictory facts.” Sorry, are you blind?
10 days ago, during a lunch on IBM event, some analyst from Gartner, and Forrester discussed this blog. The general opinion was that it is waste of time to read it. Think about this.
I will take their advise and stop to waste my time, you are not worth it.

OttoroluK

The real question here is: Is there still someone out there listening to this HDS-Fan Moron?...

Joshkrischer

Brave OttoroluK, who are you? How dare you to write this without identifying yourself. What a "bravery" and deep insights. I will be happy to challenge you on everything.
Barry, you may shame yourself to publish this comment. We were never friends but I always had some appreciation for you. It’s gone.

the storage anarchist

Josh, I rarely censor comments on my blog, and never when they are voicing a person's opinion.

Just as you are entitled to voice your opinions (and thus I did not censor them), so too is anyone else entitled to voice theirs.

Joshkrischer

Just for the records:
OttoroLuk is Stefano Panigada

Advisory Technical Consultant at EMC in, Milan Area, Italy

the storage anarchist

If you say so...

OttoroluK

Josh,
My comment did not mean to offend you, my apologies, your insistence, however, is sometimes irritating; everyone is free to have their opinions, but to question the opinions of others is not always correct. I'm sure you would have much to teach, but I think a good teacher, first of all, should be able to listen to others.

All the best.

Joshkrischer

Stefano,
I am listening to everyone but as an analyst I have to build my opinion based on available and filtered information.
I can quote a much respected analyst which wrote (un-solicited) in LinkedIn:
Andy Butler
Vice President and Distinguished Analyst , Gartner Inc.
“Josh Krischer demonstrates a knowledge of the server and storage markets that is nothing less than encyclopedic. He is blunt, honest, uncompromising and often irascible - but very rarely wrong!” February 27, 2006
Will advise you to read my profile and some other testimonies in LinkedIn.

Wplein

@JoshKrischer: Cereva went belly up in mid 2002, EMC picking up the IP in July/August. I followed this closely as I was working for a competitor (Rhapsody Networks). DMX was introduced the following February.

If you think that they took the IP and folded into a product in 6 months or less, I'm dumbfounded.

alvarezjedi

@Josh et al:

I have so many problems with the spin and propaganda that Barry churns out (heck, it's EMC, what would you expect). I especially have problems with the outright misleading conjecture around XIV. One day I'll post something on this and really get into it with him (yes, I represent the XIV product).

BUT I am posting on this thread to give him some props. He didn't censor me in nasty posts prior when he easily could have -

Please visit this older post to see: http://thestorageanarchist.typepad.com/weblog/2010/06/3004-tell-them-why-an-idea-worth-spreading.html

The comments to this entry are closed.

anarchy cannot be moderated

about
the storage anarchist


View Barry Burke's profile on LinkedIn Digg Facebook FriendFeed LinkedIn Ning Other... Other... Other... Pandora Technorati Twitter Typepad YouTube

disclaimer

I am unabashedly an employee of EMC, but the opinions expressed here are entirely my own. I am a blogger who works at EMC, not an EMC blogger. This is my blog, and not EMC's. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC.

search & follow

search blogs by many emc employees:

search this blog only:

 posts feed
      Subscribe by Email
 
 comments feed
 

 visit the anarchist @home
 
follow me on twitter follow me on twitter

TwitterCounter for @storageanarchy

recommended reads

privacy policy

This blog uses Google Ads to serve relevant ads with posts & comments. Google may use DoubleClick cookies to collect information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide ads about goods and services of interest to you. If you would like more information about this practice and your options for not having this information used by Google, please visit the Google Privacy Center.

All comments and trackbacks are moderated. Courteous comments always welcomed.

Email addresses are requested for validation of comment submitters only, and will not be shared or sold.

Use OpenDNS