The art of benchmarking

Doing a benchmark will help you seize the market of the targeted service/system/tool you want to use. It will help you to understand the available offers, the top providers, get in touch with then, perhaps help you realize your forget an important feature or requirement and the most important evaluate provider between them.

At the end, you want a product answering your need for a balanced price.

The art of benchmarking

The big line of the process are the following:

1. Gather requirement and expectation
2. Put IT in the loop (always)
3. Benchmark
4. Short list the product/service/hardware
5. Testing or Proof of concept
6. Validate de cost
7. Negotiation
8. Integration

1. Defined your needs and requirements

One of the most important steps of the process. What do you need, want or achieved with this new product.

The whole purpose of a benchmark is to select something base on objective criteria. The key point here is to list everything you want with a small sentence that can be answers by yes or no.

Example:

• Is it possible to integrate your solution with SAML for SSO?
• Can I charge is with a USB-C?
• Does the house have air-conditioning?

1.1 The IT requirements

Important part because it will condition if a product is compatible with your IS/Service catalog/process. Provide your requirements to the user in charge of benchmarking products. As mention above, list your requirement with close question that can be answers by yes or no.

Example:

• Is it possible to integrate your solution with SAML for SSO?
• Can we provisioned the user from an active directory?
• Is there permission role management?

If you have a security team, ask them what are their requirements and add them to your list ;)

2. Put IT in the loop

Nowadays, most product are SaaS, and your company will start looking into different service to manage part of their business process. Ask them to put IT in the loop every time they start a benchmark.

Don’t do the job for them, but guide them along the process and the requirement validation. Explain why it’s important to be compliant, what is the return on investment.

Pro-tips, ask your procurement/finance team to also check that the requirement are there. User can by pass IT but at one point your have to pay the invoice, so finance is never skip ;)

3. The benchmark

Now, you should have the list of features/requirements that need to be covered, all should be answered by yes or no.

The goal is to score each provider to compare them. The weighting system work quite well with the closed questions.

Pro-tips: Avoid using rating system base on user feeling. You want to be as objective as possible. Today you are the one using the tool but tomorrow maybe not.

Personally, I love using a spreadsheet for this exercise. I start by listing everything that can be answered by yes or no, then I add the remaining information like pricing, license usage below. Check the example below.

3.1 How to build your spreadsheet

Open a spreadsheet and add the following column:

• Features: This column list of the feature and requirement that you need to evaluate
• Weight: Make sure that the provider you will select handle at least the main feature
• Category: If you have a long list of feature and requirement, it’s good to split them per category.
• Provider 1: The provider column will contain the answer if they can provide the requested feature. Column should contain 1 or 0 (yes or no)
• Provider 2
• Provider 3
• Provider 4
• Provider X

The row after your feature will contain the provider score. It’s a simple sum product between the weight and the provider answers (1 or 0).

feature	category	weight	provider 1	provider 2	provider 3	provider 4	provider 5	provider 6	provider 7	provider 8
f1	cat1	50	1	1	0	1	1	0	1	1
f2	cat2	50	0	0	1	1	1	1	1	1
f3	cat2	50	1	1	0	1	1	1	1	0
f4	cat2	20	0	1	0	1	1	1	1	1
f5	cat1	20	1	1	0	0	1	1		1
f6	cat2	20	0	1	0	1	1	1	0	1
f7	cat2	10	1	0	1	1	0	1	1	0
f8	cat1	10	1	1	1	1	1	1	0	1
f9	cat1	10	0	0	0	1	1	1	1	0
f10	cat3	10	1	1	1	1	0	1	1	0
Score		250	150	180	80	230	230	200	200	170
price/user (€)		120	90	70	100	100	200	75	80	65

• 1 (yes) -> the feature is available
• 0 (no) -> the feature is not supported

4. The shortlist

Depending on the market, you can end up with 5 to 10 providers. There is no point to test 10 products. Select your top 3 and stick to it.

1. Select the 3 top scores (they are made for that)
2. Check that the top feature/requirement are covered. Drill down with a pivot table under the different categories.

Example:

Item	Provider 4	Provider 5	Provider 6	Provider 7
Score	230	230	200	200
price/users	100	100	200	75

We have a tight between Provider 6 and Provider 7.

If you analyze the score per weight you will see that Provider 6 do not match one of the top requirement/feature listed, in this case we can remove it because the feature is mandatory. Weight is used for this reason.

Weight	Max	Provider 6	Provider 7
10	4	4	3
20	3	3	1
50	3	2	3

The top 3 will come easily:

Item	Provider 4	Provider 5	Provider 7
Score	230	230	200
price/users	100	100	75

Give a good look into the pricing model of each provider. Sometimes they have different way to price the product. You will have to find, as much as possible, a common ground to compare also the pricing and help your procurement team or your credit card before buying the service.

5. Testing

Today, most of the provider propose testing/trial period for their product. Used it as much as you can, ask for extension if needed. Try to evaluate at least the most import feature, make sure they behave as attended and expected. This step let you also appreciate the UX of the tool (maybe the tool cover 100% of the features, but it’s hard to used, and you can get a high reject percentage when reaching the deployment phase). It’s not mandatory to do a POC with the entire top list, but at least 2.

Furthermore, depending on the feature, feel free to have a look with IT for testing some integration like SSO, user provisioning, or any specific feature a bit for fancier than usual.

Pro-tips, do not trust provider screenshot or dataset. Use your own, make sure it will work with your data. Provider will give you data set that work 100% with the tool.

6. Validate de cost

Depending on your company or process, you may already have the expected cost in your yearly/quarterly budget, or you have to approve every spending. Nevertheless, it’s still good to revalidate the cost with your finance department. Especially if there is deviation from your initial plan. It will also help your finance team to prepare the payment and remind them that an important cost is expected soon.

7. Negotiation

Moreover, depending on your company, you may have a procurement team available for negotiation. If it’s the case, use it, all the time. If you don’t have, check if you can use an external company or do it yourself if you have the skill.

Try to negotiate as much as you can. The money saved can be used for another tool, a team building or a bonus for your team.

But don’t be lazy, prepare the ground to maximize the discount. I usually send an email or a ticket with the following information:

1. Context
   1. Method to select top 3
   2. Table with the following information
      1. Ranking of the provider from 1 to 3 (1 is the best)
      2. Quantity needed
      3. Price/unit
      4. Total expected cost/year in EUR
      5. Contact point for each provider, phone, and email
2. An expected deadline
   1. Don’t ask for the next business day. Negotiation demands some time. Provider need to be contacted and often there are several rounds of negotiation.
3. Indicate if the cost is validated by finance and affected to your budget
4. Attached the quotations from the providers to the ticket

Moreover, I also schedule a meeting to brief the team.

Example:

Hello team,

I am writing today to request your help for a contract negotiation.

#Expected work

• Contract negotiation and help for signature with a cost range between: 50 and 70k€/year

#Context

We are looking into an IdP (Identity provider) solution to centralize all access and authorization for our applications. We will need a license for each employee (2000).

#Provider choice

We’ve contacted 5 companies and evaluated their performance based on a questionnaire. From that, we have selected the 3 best ones and conducted a thorough technical testing.

Provider name	Okta	Onelogin	Fusion Auth
Ranking	1	3	2
Quantity of license	2000	2000	2000
Cost/license/year	60	48	50.4
Expected cost/year	120000	96000	100800
Contact	email + phone	email + phone	email + phone

We estimate that with the actual growing rate, we’ll have 3000 licenses in the second half of 2021.

#Budget

This project is already included and approved in the IT budget

#Deadline

• Contract signed by end of August 2020

#Attachments

• Attached quotation from all providers

With love <3

Integration

The contract is signed, it’s time to integrate the solution if your production.

When you integrate a new service, there are several departments involve.

Usually on the IT side you will look for SSO integration, license order process and documentation, user access, and so on.

Maybe finance need to be involved to pay the license increase invoice.

Maybe you have an infrastructure or platform team, and they need to create DNS, server or set up the monitoring of the service.

Remember, to address the following topics before your production deployment.

• Licenses: Who is chage of requesting the increase. To which budget ?
• Contract: Who will ahgve to sign it and keep track
• Documentation: Who is charge of the writing the documentation and its maitnenance
• Access and permission
• Support

Conclusion

Product selection is not trivial, but with the few guidelines and process aboce you will maximize the change to make the right choice for your business.