Vue Software

Archive for the 'Testing' Category

Employees Suing Sprint Over Commission Snafu

Published
by
admin
on December 10, 2008
in Best Practices, Industry News, Miscellaneous and Testing
. 2 Comments
1 Star2 Stars3 Stars4 Stars5 Stars (1 votes, average: 4 out of 5)
Loading ... Loading ...

A recent article in InformationWeek pointed out that “The wireless carrier is facing a class-action lawsuit over allegations that it shafted employees of commissions totaling more than $5 million”.  Who is the culprit?  The integration of Sprint’s and Nextel’s is supposed to have caused the system failure leading to potentially 19,000 employees not receiving their full commission on sales of new phones, calling plans and accessories.

Chris Cabrera, the founder of Xactly, actually found this article before I did, and came up with a few  conclusions.  1) This type of mess can be avoided.  2) Providing sales reps with real-time visibility in their commission plans and performance should help out identify issues before they cascade in a lawsuit.  3) Audit trails are mandatory and should make the appealing process much quicker and straightforward.

So we know that many companies using legacy commission systems or spreadsheets often make several mistakes in calculating their employee’s commission.  However, Sprint IS using one of the leading compensation solutions.  This means employees probably have real-time visibility in their commission and that audit trails are available.  It also means that any challenges related to contesting an incorrect commission is probably caused by internal processes or the lack of resources rather than a lack of functionality in the comp solution.

According to the article, the integration of back-end systems is to be blamed.  This goes back to several of my earlier posts regarding how sales performance systems were not a silver bullet to fix all comp related problems.  If the correct information is not fed to the comp system, the results won’t be accurate.   The other hint pointing in another direction than that of the comp system is that Sprint spent $10 million to fix the problem.  That’s a lot of money.  That’s much more than the cost of implementing a brand new leading SPM solution…

Could this situation have been avoided easily?  Probably not easily, but I’m sure it could have been avoided.  How?  By more rigorous  testing of the backend systems, by more rigorous testing of the end-to-end comp process, and probably by having more complete test data and better defined expected results.  Too often do I see a situation where an implementation is tested very well in isolation (unit tests, and system tests), but where the end-to-end tests (system integration tests, user acceptance tests) could use more attention.  Once an issue is identified, it should be relatively “simple” to fix it.  This situation shows the importance of paying incentive compensation accurately, which can only be achieved by identifying defects before the system goes “live”.

Tags: , , , , , , , ,

Related Posts:
The 12 Days of Christmas Incentive Compensation Blues
Team Performance Poll

SPM Testing Template - Part 5

Published
by
Julien Dionne
on May 23, 2008
in Testing
. 0 Comments
1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

Incentive Compensation Management test results can be recorded in several ways. One of the approach I like to use groups all plan names, rule names, conditions, expected results and testing status on the same spreadsheet. I find that by keeping all this information together, it is easier to quickly get the picture of the overall testing progress. It also allows to keep all the information on the same spreadsheet instead of having to maintain 2 spreadsheets with identical information.

Here is an example to get started:

Other benefits of using an Excel spreadsheet to record test results instead of a Word document include:

  • Ability to quickly highlight failing tests in red
  • Ability to filter information displayed (e.g. display only failing tests)
  • Test results can be printed on fewer pages
  • Ability to create macros to perform more “fancy” features such as displaying the number of days a certain issue has been opened.

Other columns could be added to add additional information such as the date at which the test was performed, the name of the tester, how critical the test is, the actual result when different from the expected result, and comments.

Tags: , , , ,

Related Posts:
Resource Center
YAWN… ICM Testing – Part 1: Introduction

ICM Testing - Reviewing Results - Part 4

Published
by
Julien Dionne
on May 20, 2008
in Testing
. 0 Comments
1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

When test scenarios for an SPM application are defined properly, reviewing the results should be straightforward. However because of the volume of results typically generated, finding the correct result can be a very time consuming process without the proper skills (finding output of rule A in the 1 million results generated can be like looking for a needle in a hay stack, except that during testing, there are several needles that need to be found!).

Every ICM application I have implemented offer an interface to browse through the results. This is usually a GREAT tool to troubleshoot and to resolve issues, but it’s rarely the best and quickest tool to analyze results in bulk. Fortunately, every ICM application I have seen also have a functionality to export the results. Exporting the results will create a file locally which can then be opened by a program such as Excel or Access for further analysis. I will discuss how to use Excel to review the results, and later I will create another quick tutorial on using Access.

Step 1) Export the Results from your ICM System
Every system has its own way of doing this, but generally this should be doable from the interface and not requiring a database administrator to pull the data from the database (although this can also be done if required).

Step 2) Import the Data in Excel
First, open Excel and click on File->Open. Click on the dropdown menu beside “Files of type” and click on “All Files”.
Find the location where you saved the exported file of step 1 and click on “Open”.

If the file is a .CSV file, Excel will automatically place the data in columns. If the file is a .TXT file, more steps are required (see step 3).

Step 3: Opening a Delimited Text File in Excel
In the first Wizard step, choose “Delimited” and click on “Next”. In Step 2, select the character used to delimit each column (Xactly uses semi-colons). Click “Next” twice and the data will be imported in columns.


Step 4: Filtering the Result Data
At this point we have a worksheet with all of our result set displayed. Each of the column can be sorted and the search feature can be used to quickly find a specific order. However there is another trick which allows to quickly display data for a specific rule, result name, person, date, etc.

Click on Date->Filter->AutoFilter

This will make the first row (usually column names) dropdown menus which will contain all the values contained within the column. In the example below, the drop-down for column “Currency” allows to choose “USD” and “YEN”. Choosing USD will filter out all the other currency types.

Filtering can be done in this manner for multiple columns at the same time. For example you could filter a certain position (Julien Dionne), and further filter to display only the USD currency.

Quick Summation
Summing credit and commission amounts is one of the most frequent activity I perform when testing.

Excel has a built in feature which allows to quickly find out the sum of selected cells. By simply highlighting the cells containing a number or a dollar amount, the total value of these cells will be displayed in the bottom-right corner of the window.

Additional Notes

Excel offers more advanced functionalities such as Pivot Tables. They can be useful when reviewing a large volume of results. I will discuss this in more details in the future.

Excel Limitations

Excel cannot have more than 65536 rows. If the result set exceeds this, it will not get imported. It is possible to “workaround” this issue by only importing a subset of data (maybe by restricting the incentive period). MS Access could also be used instead of Excel and would not be subject to this limit. I will discuss using Access to review results in the near future.

Tags: , , , , ,

Related Posts:
Resource Center
YAWN… ICM Testing – Part 1: Introduction

ICM Test Planning, Scenarios and Templates - Part 3

Published
by
Julien Dionne
on May 16, 2008
in Testing
. 0 Comments
1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

In my last post about testing I said that ICM / SPM systems should be tested in phases; the reason for this is that discovering issues late in the development life-cycle could add unexpected delays and ultimately make the budget run over-budget.

It is surprising how often I have seen the business users and stakeholders not agree with the results and the development team exclaim, “Oh, that’s how it’s supposed to work!” or “That’s what you meant!”. Without proper planning, it is possible that during UAT the users will try to perform various activities and “break” the system - to which the development team usually answers “You’re not supposed to do that”, or “That’s not how it’s supposed to work”. Often, shortly before the payroll date, business users often ask questions such as “What about the draws?” or “What if orders have negative amounts” (good requirement documentation is also important!). Proper planning should eliminate all those nasty surprises.
There is an excellent article on Wikipedia regarding creating a test plan based on the IEEE 829 format. The test plan describes what will be tested, how it will be tested, what will be the deliverables, who will be responsible for what, etc.

Creating Test Scenarios for a Sales Performance Management System

I believe the most important aspect of testing is the test scenario preparation. I briefly mentioned how creating good test scenarios was particularly difficult with an ICM application because of the volume of test cases it will typically generate. This is unavoidable, but proper planning is required to ensure that tests are not testing the same conditions twice (wasting time) and that all conditions are being tested (not cutting corners).

A test scenario should have a name, a scenario ID and a description. This will help quickly refer to them during meetings. The test scenario should include the initial conditions, input (such as the order type), and the outcome or expected results. Finally, it is a good practice to list the business requirement ID that the scenario is testing. It is important for the test scenarios cover each plan, each rule, and each formula used within the rules. To test an ICM system, I like to group the scenarios, by Plan and by Rule:
1. Plan A
1.1 Rule A
1.1.1 Scenario 1
1.1.2 Scenario 2
1.1.3 Scenario 3
1.2 Rule B

…Creating Test Data
The test data is the data that will be “staged” to test the scenarios. Typically, an order or a combination of orders will be required to test different scenarios. These orders should be created in the appropriate format to be staged and it should be documented which orders test which scenarios. Because of the different testing phases and because test data is often altered or corrupted during testing, it is important for the test data files to be kept together and be readily available to be re-staged when required.

After processing all the test data, all the test scenarios should be tested. That should cover the entire system, and in theory, after completing these steps, the system should have no outstanding defect.

Tags: , , , ,

Related Posts:
Resource Center
6 Phases of the Sales Performance Management Delivery Model

ICM Testing – Part 2: Test Phases and Traps

Published
by
Julien Dionne
on May 9, 2008
in Testing
. 2 Comments
1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...
ICM systems, as every other software system, should be tested in phases. Here are the major test phases (there can be more or less depending on the complexity and size of the project)

Unit Test: Performed by the developer to ensure a specific component performs as per the requirements. Unit tests should be performed as the system is being built.

System Test (ST): While unit testing was performed with only a single rule, system testing tests entire plans. It ensures the entire system performs as expected and as defined.

System Integration Test (SIT): An ICM system often relies on receiving data from other existing systems. For example, it may expect a file received via FTP, from an ETL process or fetch data directly from a CRM system such as SAP or Peoplesoft. SIT will ensure the systems interact the way they should.

User Acceptance Testing (UAT): Once the ICM system is fully tested by the implementers, UAT is when the business user or owner of the system test and review the results of the system. UAT should be performed on the production system, with production-like data.

Other very important tests which should be performed during ST, SIT and UAT include:

Performance Testing: This test confirms if the processing of commissions can be performed within the required window.

Stress Testing: This test verifies how the system behaves if the data load is higher than expected. For example, the compensation data may increase year over year, or may be higher during quarter end, year end.

Regression Testing: Every time a system is altered, there is a risk for a defect to be introduced. Relevant tests should be performed again to ensure the system still works as expected.

Each of those test phases are usually done in parellel to major development life cycle activities - the “V-model” above illustrates this. I will have to keep the development phases discussion for the future but yes, the left part of the “V” is also necessary.

When working with an implementation partner or vendor, the client is usually mostly responsible for the User Acceptance Testing which I will discuss in more details later.

What I must stress as being extremely important is that it cannot be taken for granted that the implementer will necessarily have a rigorous testing methodology and ethics. To avoid having big surprises when reaching UAT, the client should keep an eye on other testing activities and ask for regular status updates.

Another common trap is that getting regular status updates may not be enough. It is important for the project team to design and agree on the test cases and test plan early on. In larger implementations this is often a step which should get a sign-off. This will ensure the implementers know what the expected results should be, which is very different than the implementers assuming they know what these results should be.

Tags: , ,

Related Posts:
YAWN… ICM Testing – Part 1: Introduction
Resource Center

YAWN… ICM Testing – Part 1: Introduction

Published
by
Julien Dionne
on May 8, 2008
in Testing
. 0 Comments
1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading ... Loading ...

Testing is a quality assurance process required to ensure the system works as expected and as defined. I never met anyone who found testing “sexy”, fun, or even interesting. However, it is essential!

No matter how well designed the compensation plans are, if the commission and bonus results are not correct, the implementation will be a failure. We discussed that it is important for payees to be able to correlate their rewards to their performance, but mistakes could prevent them from doing this efficiently. Another result of “bugs” or system defects will be an increased number of complaints from the payees which will consume the compensation team’s time. These errors will also add confusion, making sales people unhappy and ultimatly lead to a lost of confidence in the system.

Traditionally, testing was a task done only once the entire implementation was completed; you built the system, you tested it and you rolled it out. This is one of the best recipes for failure. Test planning should be done at the very beginning of the project and testing performed throughout the development of the system, until it is rolled out.

Testing an ICM system is particularly challenging because of the volume of different cases. A “standard” software system testing may have a test that if a button is clicked, an action will be performed. With an ICM system, every plan, every rule, and every formula should be tested. For example, if there are four quarters, each quarter using its own rate table, each rate table having 8 tiers, and each tier corresponding to a formula, there will be 32 individual formulas to test – 32 test cases.

In my next posts I will discuss many testing related topics including the different test phases, how to create proper test cases and test plans, and how to review results in Excel and in Access.

Tags: , ,

Related Posts:
Resource Center
ICM Testing – Part 2: Test Phases and Traps