Does any business looking at its ICT systems really need to be told to ‘Try before You Buy’? Is a demonstration accompanied by a one-page proposal summary really enough? On the other hand, there are people who believe, or at least suspect, that full-blown test installations cost a disproportionate share of the overall investment and tend to uncover only techy tweaks that would have surfaced at installation time anyway. In our thoroughly regulated world of today there is more than a suspicion that many systems evaluation exercises are in truth exercises in cladding the fundament so that any subsequent and adverse consequences, personal, professional or corporate, will be met by an appropriate volume of test results.
Software is of course the main ICT area where evaluation and testing arises. In general it is because software in real world use has always had a tendency to throw up some kinks that were not anticipated by the developers. Many of those, especially the old pros, might say that there are always elements that could not be foreseen because there is actually no depth of stupidity that users en masse will not explore over time. Infrastructure, on the other hand, from servers and SANs to WANs to MANs has to be judged on performance specifications. That is, in fact, usually sufficient because the really technical testing has been done at the pre-manufacture stage by the vendors and tested again by significant partners and even competitors. Software, on the other hand, whether released fully into the wild world of the Web and consumers or corralled in a corporation, is considerably less predictable as real life rubs against design and theory.
Virtualised
When evaluation and testing are discussed today, virtualisation is a term that is guaranteed to occur in the first few sentences. That will be partly because virtualisation projects are high on all lists of evaluation exercises in recent years and partly because virtual environments are seen as ideal and economic platforms on which to test new or changed systems. Douglas Philips is a senior marketing manager in VMware Europe, a virtualisation veteran who came to VMware with its purchase of Akimbi, a software developer of systems to build a test infrastructure to automate the setup and teardown of multiple virtual machine environments. Its technology is now part of VMware vCenter Lab Manager, which is used to manage cloned and temporary virtual environments.
“Lab Manager is purpose built for software test and development and incorporates a huge range of experience. So if you are evaluating a new version of some enterprise software, for example, it is possible to replicate every server and client that will connect with that system.” In the hardware world, provisioning all of those would be almost impossible and prone to error, as anyone can understand. But Philips is happy to acknowledge that a similar test in a virtualised environment is not infallible. “There are likely to be anomalies, even the ‘ghost in the machine’ effect that means that you can replicate something 999 times exactly and then a kink with no visible cause develops. It is very similar to the bug in an application that is difficult to track down because it is almost impossible to replicate the exact circumstances in which it shows up.”
Today, he claims, Lab Manager overcomes most of these and other problems by providing a safe test and development environment replicated from the production systems and also providing security in regard to what data developers (in-house or external) can see or use. “Most importantly, it can manage through the various stages of software deployment from early stage testing and quality assurance through pilot tests on, for example, new hardware or other platform variations. You can take the process through to user acceptance trials before promoting to the production systems.”
In regard to server virtualisation, Philips says, more and more organisations are simply moving directly into provisioning virtual machines for production. Where there is evaluation it is often directed simply towards proving to the CIO and the C-suite generally that performance will be the same or better. “With applications, there are still a lot of instances where virtualisation is piloted successfully, say in one business unit, and then stalls. Often it is because control of budgets is spread and possibly because IT is just not selling the advantages strongly enough internally. On the other hand, there are already many larger organisations where the rule is ‘virtual only’ and a business case needs to be made for a dedicated physical asset.”
Independence
When larger organisations take the evaluation and testing of proposed new systems or significant change seriously they are likely to turn to specialist consultants. Software Quality Systems or SQS has a 27 year track record as a leading independent consultancy in software testing. Cologne-headquartered it has an established branch in Ireland headed by managing director Phil Codd. “Our first role is to be independent. That means that before we get into anything technical we have to work with the client to ensure that there is a clear and defined understanding of what the organisation wants to do and what is required of the new system. Any evaluation has to be against those criteria.”
Where strategic business is involved the definition of requirements really has to be copper bottomed, he says. “If the definitions are loose the solution will be loose, invariably. That may not lead to ‘failure’ but it certainly means that the new systems are unlikely to fit the real needs of the organisation as closely as they should and further drift in the future is quite likely. Our service is to impartially evaluate the clarity, completeness and lack of ambiguity in the requirements definition.” If an organisation issues a tender document that contains ambiguity, Codd points out, suppliers will, not unnaturally, respond as they see fit, as their interpretation suggests. One consequence will almost certainly be changes during the process, one of the most common causes of cost over-runs.
He emphasises that evaluation is not confined to the technology and the software code. “There are areas of business risk that should be considered, especially since organisations are almost always looking to forge a longer term relationship with vendors of durable software. That might mean that the clever solution from a two-man band needs careful consideration, or being the first and only user in Ireland of something that is apparently successful in the US market. There are also less tangible but important questions about the cultural fit of new systems and the impact of change in the organisations. Another common situation is when organisations are being integrated after merger or acquisition.”
Technology has matured and there are clearer standards and better defined rules today, for the interoperability of software, for example. “That is not quite as clear cut in the evaluation process,” Codd points out. “We certainly have proven evaluation methodology, from business requirement to line by line testing of code. Our function for our clients is to take as much as possible of the risk out of investing in new systems.”
Power and space
Another active testing service is run by Dell Ireland in its Limerick Solutions Lab which works for clients all over EMEA as well as a global demonstration capability for Dell itself. “We have about 300 servers and 100 Terabytes of data storage to play with,” says manager Anthony Quigney. Like SQS, the Dell Solutions Lab has developed a range of methodology in-house which it brings to bear on each customer’s requirement, whether evaluating new architecture-including virtualisation and cloud computing-or undertaking a ‘shoot-out’ comparison of proposed software solutions to defined business needs.
“The range of resources means we can accurately simulate the customer’s existing or proposed systems. Using real sample data is standard, for example, but under strict guideless and usually anonymised and fully secured and in fact scrubbed afterwards to agreed standards. In that way we can test a full set of applications, end to end and on different end user devices and the Web and then all the way back up the stack.” Any exercise in technology migration for an organisation is likely to be a significant investment in proportion to its own scale, Quigney points out, adding that for just that reason Dell works with relatively small organisations with special and mission-critical IT requirements as well as large corporations and public bodies.
But it is not all about testing and benchmarking, he stresses. “We work with the client and often the ISVs involved ensuring that when the technology stack comes out of the labs and into the production environment it functions exactly as designed. We are constantly trying to understand the behaviour of vertical application stacks and to fine tune and optimise them for real life working. So it is not just the architectural overview but the implementation and the benchmarks and documentation to support it. Sometimes that comes down to ensuring that the new build does in fact replicate exactly what was tested!”
Decision support
Redstone Technology has a pedigree going back to 1984. Its professional services are focussed on business critical infrastructure including server solutions and data storage. “We are asked to help our clients make significant ICT decisions and by and large we use proven project management disciplines and methodologies to evaluate,” says Eoin Mullan, senior account manager. “Broadly speaking, even in today’s economy we most often come across a rush to new hardware to meet new applications or wider deployment without checking or providing the infrastructure required. Funnily enough, we also see a bit of a rush to infrastructure because it is assumed to be a requirement. Either way, there is still a tendency to purchase equipment to underpin new or changed systems, which was what led to the problems of server sprawl. That is by no means entirely in the past.”
Like the others, Redstone lays great emphasis on establishing the business requirements accurately in advance of any evaluation. “It is critical, in our view, to get to an outline statement that is fully understood and agreed,” Mullan says. “One key is to ask the client ‘What will make this project a success?’ We certainly have to establish what is generating the business requirement and who are the stakeholders. Then we can have confidence in proceeding to the technology evaluation.”
The next step is usually proof of concept in a small scale and controlled environment, he says, which sets the benchmarks. “Yes, there is cost involved but in essence it is investment in preventing error, assurance or insurance as it were. As a project proceeds further there may recommendations by us for changes in the technology or indeed even the brief. Awareness of the technology roadmap for key vendors has been useful for us on occasions and we have, for example, recommended putting a project on temporary hold to gain significant advantages anticipated from a new generation of technology.”
Disciplined
In some respects the investment in formal evaluation of proposed system change combines a best practice approach with the old fashioned prudence of managers in larger organisations, says Rowan O’Donoghue, director of innovation and development in award-winning IT consultants Origina. “There is pressure on individuals and IT departments to demonstrate good decisions, which is by no means a bad thing. We have certainly seen in the past the kind of over-complicated sprawl that can result from cumulative investment without clearly defined requirements and objectives. What typically happens is that the cracks get bigger. Failure to recover after a disaster is often because the systems sprawl has just been too massive. They can’t be put back together again.”
So it is good that organisations go through systematic proof of concept, simulation and testing. “Sometimes that may be motivated by a corporate wish to prove impartiality, to be seen to have gone through such a process. So be it, the discipline is still important in making good decisions and systems choices.”
But another side O’Donoghue sees is that many IT departments are doubly pressed because they have not got the budget to properly maintain the inherited sprawl yet they are under pressure to meet new demands. “In that context, we have a number of information priorities we follow before we even look at the specifics of technology to be evaluated. We need to know how the organisation is using its existing technology right now. Then we want to establish a clear functional list of what the client actually needs. It is also important to identify the sponsor or driver for any project and get to an agreed requirements specification.”
In today’s market and the world of ICT generally, O’Donoghue believes also that it is a mistake to look too far ahead or invest on the basis of what might happen or be required in the future. He is not too keen on the idea of future-proofing as an IT investment strategy in business. “Right now, there is a lot to be said for focussing firmly on the absolute essentials and a short term, realistic horizon.”
Subscribers 0
Fans 0
Followers 0
Followers