Make the right people evaluate the right ideas

Nosco · Process & Evaluation Features, 2018

When you work with innovation and ideas at scale in a large organization, it can be hard to keep the overview and find the best ideas. So you team up with some colleagues and experts that can help you with the process of finding exactly those ideas that can make an impact and radical change.

Innovation processes can take many shapes, but a typical one we work with at Nosco, usually follows a classic funnel shape: It starts with a broad range of ideas and gradually refines and selects from among them, creating a handful of more tangible innovation projects that can be pushed further on in the organization for completion.

Along the way down the funnel, you have certain “gates” where ideas are evaluated by domain experts and only the best ones move on in the process. Out of thousands of ideas you end up with 1-5 ideas that are matured and ready to be turned into real projects.


Screening 01


Screening 02



Before this project, we didn’t really have any smart way of supporting this workflow on the Nosco Platform. We couldn’t really visualize the process for the users, often leaving them in the dark about what’s happening with their ideas and we weren’t able to distribute ideas to the right experts in a seamless way, which caused clients and our innovation consultants to do a lot of manual work and Excel-magic.

Since we work very close with our software clients and users on an almost daily basis and due to the fact that we often implement large scale innovation processes at some of the World’s largest organizations, we know from first-hand how difficult managing thousands of ideas and users can be.

The Value Proposition Canvas Using the Value Proposition Canvas Workshop

In 2018 we decided that it was time to improve our platform's ability to support innovation managers in facilitating their innovation processes. I prepared a range of initial user interviews to better understand how they were working — not only with our existing platform features, but also to learn about their daily routines and challenges.

A key tool in the research phase was the Value Proposition Canvas (here’s a great piece on how it can be applied in UX research). The goal of creating the canvas is to learn how to match your product features and the value you offer to your customers by understanding the users’ “jobs to be done”, “pains” and “gains”. Besides the value proposition canvas and interviews we also had a look at countless process diagrams from clients and it turned out that most of them somehow follow the same funnel-like process as described above.

After the interviews, the team got together in two workshops to review the information and what we’ve learnt so far. This also helped us to align on the scope of the feature, and we quickly realized that we were in for a massive change to the platform. We identified two main user types that we wanted to cater for: the evaluator and the innovation manager.

The scope ballooned way too much, but I think it has to do with the excitement and the “if we just could build that”-feeling the research created. Looking in the rearview mirror, I should have paid more attention to our own sales and business stakeholders, sticking to tangible and incremental improvements instead of a big overhaul.

I wanted to keep our SCRUM process tight and the developers close to ensure agile development and frequent releases. So to keep the developers close and to kickstart the project, I invited the development team to my house for an intense day of reviewing flows in InVision, discussing data structures, barbequing and beers. We also had a new guy on the development team, so the day also served as an good way to introduce him to the team and product.

BBQ in Sorø
Keep your friends close, but your developers closer.

I used this project for an opportunity to design directly in the browser by coding the initial HTML and CSS. This ensured an easy hand-over to the developers of some of the main areas of the features, and they were thrilled not to have to do this work. While the developers did the heavy lifting on the backend, me and the designers tested and designed flows, screens and new components for the next part of the feature. This way we formed rhythm of being divergent and convergent together with our developers and consultants.

Stages & Evaluation Teams

During the research and ideation phase of this project, we identified two new feature concepts for the platform: Stages and Evaluation Teams.

We knew right from the beginning that we had to support the classic funnel-like innovation process. The solution was to introduce the ability for the innovation manager to split up an innovation challenge in multiple stages: All ideas are submitted in the first stage, they are then distributed to the evaluators for expert assessment. The best ones move forward to the next stage, and so on. Much like a hiring process where you have to find the best candidate for the job.

The process are managed using tabs, a classic design pattern often used to show a chronological sequence.

We designed a classic “tab view” as it shows the chronological order of the different stages well. Each tab contain all the settings for the stage. The fixed tab bar allows the innovation manager to change to a different stage even though the page has been scrolled. This ensures overview and a seamless workflow. The innovation manager can match any existing corporate process by setting up and configuring as many stages as needed. Each stage has its own settings, complete with form builders, deadlines, permissions, etc.

To create a coherent and repeatable pattern for all the many settings available for a stage, we introduced a new “settings” component. The component had to communicate its function clearly and should be able to adapt to different actions, i.e. showing a date picker, a dropdown, a switch, buttons and more.

Recognizable icon
Intuitive call to action
Short help text

Keeping track

Another important job for the innovation manager is to keep track on all of the ideas. They need to be assessed by experts, and the experts need to be reminded of doing so. But more importantly, the innovation manager needs to be able to identify the best ideas, so they can move on in the process.

That’s why we created a new view, showing ideas and their evaluation scores. Here the innovation manager can easily compare multiple ideas, follow up on the individual expert scores and remind experts to evaluate.

Evaluation Teams

Innovation managers also told us that they used to struggle with assigning ideas to the right evaluators. Previously, an evaluator was assigned all ideas in a challenge which doesn’t really work as there could be over 2000 ideas submitted in a challenge.

That’s why we introduced the concept of Evaluation Teams: For each stage in a challenge, innovation managers can create an unlimited amount of evaluations teams, consisting of domain experts and relevant ideas. Evaluators can be found by searching for expertise and other user attributes by tapping into the client’s Active Directory.

My Evaluations

We learned that the evaluators have limited time and very little knowledge of the platform, so we had to design for a simpler experience with a focus on getting the job done.

Evaluators will now see a list of all the evaluation teams they’re a part of in the left navigation. A red dot will show how many ideas they have left to evaluate. This notification-style UI element follow an established and recognized design pattern, giving the evaluator a quick overview on how many ideas he has left to evaluate.

A large list show the ideas that have been assigned to the evaluator and their scores. The ideas can be evaluated on the go directly from the list, but they can also be opened and read using the Idea Window. Evaluators can also have private discussions with other experts about each idea to ensure that the right scores are given.

Process, process, process

Keeping users aware of the process and the current state of the innovation challenge, are also key objectives for the innovation manager. We introduced a simple timeline illustration of the process for every innovation challenge. Hovering over a step in the process will show additional information about that particular part of the process.

The results

It’s been a little more than a year since we released this set of features, and we can now see that our Customer Success team and innovation consultants spend much less time helping clients out by doing manual work and hacks on the platform (we track that using Harvest and Zendesk). We’ve also seen a huge increase in clients using the evaluation features and managing the process of their innovation challenges on their own. The user feedback has been really good and constructive, suggesting a lot of improvements and additions. One client suggested to do a heat map to identify the best ideas, which is definitely a thing I’d love to build as it would look really cool, but also serve as an awesome way to discover ideas 🔥

Other cases

A Friendly Filter

A Friendly Filter

The feedback was clear: Cients needed a way to track engagement and reach out to people who are less engaged or people who need to do certain things. A design sprint kick-started the design process.

Read more
Product Design 2019
A better way to browse ideas

A better way to browse ideas

If you’re an administrator on a Nosco platform, chances are that you are managing thousands of ideas on a daily basis. For that you need a good overview and a seamless way to browse ideas.

Read more
Product Design 2016
Nosco Rebranding

Nosco Rebranding

In 2018 it was time for Nosco to get a new self portrait. That's why we teamed up with the extremely talented branding agency AM from Copenhagen.

Read more
Branding 2019
How a design system, principles, and tooling shape a company

How a design system, principles, and tooling shape a company

For some time, I have been quite inspired by design systems, and with the addition of new team members and a growing business, developing a proper design system just made even more sense.

Read more
Design Teams 2018 2019