Copyright © 1998, Nanette Miner
Permission is granted to reproduce or excerpt this article if proper credit is given in this format:
Reprinted/excerpted with permission. The Training Doctor. Author: Nanette Miner. www.trainingdr.com

Successes, and Failures, of a Management Development Program
A Case Study

By Nanette Miner, EdD

A Case Study

Ever wonder if the management development programs you're running are making a difference? Making a difference in people's skills, business outcomes, bottom line dollar results? In 1999 a small east-coast retailer wondered the same things - and set out to determine if training was truly making an impact on the organization.

Background

The company was founded as a one-store mom and pop surplus goods retailer in 1952. The company experienced slow, steady growth during its first three and a half decades, operating five stores by 1990, all in its home state. In 1990, the company was purchased by a major retailing corporation and experienced explosive growth over the next seven years, growing to 36 stores in seven states. By 1996 each store had up to 18 managers. In 1996 the company experienced a reorganization, closing seven underperforming stores and eliminating in-store levels of management in the remaining stores. During the period of rapid expansion most training focused on operations-type training; that is, how to physically run a store. The extent of formality surrounding the training program consisted of a one-page checklist that simply listed the departments and operational functions that needed to be reviewed. New manager training was carried out by experienced store managers. The extent of training was entirely dependent on the knowledge of the trainer (store manager) who was conducting the training. The format and quality of training fluctuated from store to store depending on the trainer, his/her familiarity with each topic, and his/her tenure with the company. Nearly all new store managers were recruited from other retail organizations with the assumption that they came with management skills and simply needed to learn how to operate a store in this environment.

In September of 1995 a new training director - the same director that is in place now -joined the company. Prior to her joining the company, the position of training director had been vacant for nearly a year. During the reorganization the training department was reduced from four to two, leaving a rather limited staff to carry out a tremendous initiative.

Issues

During the first few months of her tenure, the new training director concentrated on a needs analysis of the training situation, the population, and the company's future needs. A number of significant issues were exposed:

Inconsistency and lack of formality in training
Some training in managerial topics, such as giving reviews, progressive discipline, and conducting interviews was given by in-store HR Managers. In order to teach one of these training program, the HR Manager would first attend the session delivered by the training department at the home office. Each in-store HR Manager would then bring the training back to his/her stores and deliver it on an as-needed basis as determined by that HR Manager. The HR managers were not monitored or held accountable for actually conducting the training in the field, nor was there any tracking of who attended or whether or not the training actually translated to work-behavior.

Lack of bench-strength
It was determined that, traditionally, few managers were promoted from within the company and that the company’s bench strength--that is, those people who would presumably advance from lower levels to higher levels of management--would need to be strengthened, or the company would continually need to recruit managers from outside the organization - an expensive proposition.

Turnover
The Saratoga Institute of Santa Clara, California, was brought in by the Human Resources VP to determine the way in which HR (and training) could best become an asset to the company. Although the company had lower-than-average turnover, for a retail organization, it was still over 100% per year and was costing the company significant money for recruitment, hiring, orientation, learning curve, etc. The Saratoga Institute’s recommendation was that the singlemost significant contribution that the Human Resources department could make would be to lower turnover at the associate level (hourly employees).

It was hoped that a formalized plan and implementation of training and management development would resolve these issues.

Formalizing the Training Plan

The management development initiative was planned throughout 1996. First, an analysis was done of the 'best of the best' managers that existed within the organization; how they spent their time, how they led, how they managed, task analysis, and work flow processes were all analyzed. From the analysis, a group of competencies were developed which gave the training department something to shoot for when designing the management development program. The competencies included: commitment to results, customer focus, setting priorities, managerial courage, selection and development, team building, and communication and resolution.

Using the competencies as a foundation, a core group of nine courses were either developed in-house, contracted out, or purchased off-the-shelf. The courses included: Coaching, Communicating to Manage Performance, Interviewing & Hiring, Giving Reviews, Progressive Discipline, The Counseling Process, Preventing Harassment, Sexual Harassment (required by the state), Recruiting.

The management development program was rolled out during 1997 and 1998, with delivery via classroom instruction, self-study, video or audio.

Expectations and Assumptions
A number of assumptions were made in regard to the impact that the training would have, both on the management and the associate population. These assumptions, it turned out, would lead the evaluation of the program astray a few times. It was expected that by giving store managers increased knowledge and skills in the field of management, turnover would be reduced and the incumbent store management would be better qualified to fill future management demands within the company. More specifically, it was assumed that a well-trained manager would feel more competent and fulfilled in his/her position and thus stay with the organization longer. It was also believed that a trickle down effect would occur once management was trained, meaning that a well-trained manager would create a better working culture within his/her store and thus turnover at the associate level would also be reduced.

After two years of management development training (1997 and 1998), a comparison of pre-training business results (1996) and post-training business results was undertaken to determine if this tremendous endeavor had an impact on the organization. The news was both good and bad - and in many ways surprising.

Evaluation

The topic of training evaluation is a difficult one, at best. You'll find a whole cadre of training professionals who say that evaluating training just can't be done. There are too many variables such as what happens when the trainees get back on-the-job, whether or not management will allow for practice or will reinforce the new skills and behaviors, and whether there are other factors outside of the reach of training that will be affecting the trainees behaviors, perhaps counterbalancing all the good the training did.

While the above named factors may make it difficult to be 100% precise in our calculations, it is possible to collect mitigating evidence in favor of the effects of training. In the case of our retailing organization, four criteria were selected by which to judge the success or failure of the management development program. They were,

1. management turnover
2. management promotion from within
3. associate turnover
4. associate retention

One of the mistakes made in the evaluation of this particular program was that the decision to evaluate was made rather late in the process. It is wise to plan for the evaluation of a program during the design phase, so that appropriate data is collected. In our case it was impossible to attach strict monetary figures to each of the criteria because the HRIS record keeping process changed midway through the project and pre-training and post-training data was not always codified in the same manner.

Management Turnover
Management turnover decreased by 28% over the course of the three years. Both voluntary and involuntary terminations were tracked. The involuntary terminations decreased significantly. This might be a result of better training and increased management skills translating into managers who know how to do their jobs better.

Management Promotion From Within
Along the same lines as the reduction in management turnover, management promotions from within also increased steadily. In 1997, 20% of management positions were filled from within the organization; in 1998, 25% of management positions were filled from within the organization.

Typically, 54 management positions had been filled per year from outside of the company. The calculated cost of recruitment, orientation, and learning curve for each new manager was $2585.00. Over the course of 1997 and 1998, 129 managers were promoted from within - saving the company an estimated $333,465.00.

Associate Turnover
Associate turnover was one of the areas that held a surprising outcome. The rate of associate turnover remained steady over the course of the three years (including pre-training and post-training). One of the presumptions made at the outset of the training initiative was that there would be a trickle down effect. That is, if training was applied at the management level that increase manager's skills and made the workplace more enjoyable, then associate turnover would respond positively. The lesson learned was that in order to affect a population - one needs to apply the training to that population. The Saratoga Institute had recommended that a reduction in associate turnover be the goal and yet training was not applied to this group and no change in the level of turnover occurred during the three years of the study.

Associate Retention
Associate retention had to be looked at closely. At first glance it appeared that associate retention would not have changed if the rate of associate turnover had not. How could retention be any better if the organization were still turning over 2000+ associates a year? Yet, upon closer examination it was discovered that overall, the length of retention had increased significantly. Retention was categorized in one of five ways: less than 30 days employment, 30 days employment, 60 days employment, 90 days employment, and more than 90 days employment but less than 12 months. During 1996, most associates left the organization within the first 30 days of employment. During 1997 and 1998, most new employees were staying between 60 and 90 days. Presumably this resulted in a cost savings in terms of a recouping of the money spent to hire, orient, and train a new associate. But those numbers were not identified or calculated.

Overall, however, associate retention and turnover, which was a large motivator of the management development project, had no beneficial outcome.

Lessons Learned

The evaluation of this training project exposed some important lessons that are applicable to all training initiatives and evaluation projects.

• Know what behaviors or outcomes you're aiming for - so that all of your training initiatives have a common focus and purpose.
• Before you even begin the training - spend time determining how you're going to evaluate it. Are you looking for true Return on Investment or is a Benefit Cost Ratio analysis sufficient? Are you looking for a dollar return or simply a change in performance and/or behavior (which more than likely will result in a beneficial monetary outcome - but that will be a lot harder to prove)?
• Hand in hand with determining how you are going to evaluate the training, spend time up-front determining what data you will collect and how readily available it is. This is essential in a large-scale project that may encompass many years because you want to be sure that during the evaluation phase you are comparing like-with-like. In the case of this project, the HRIS record keeping system was changed midway through the training initiative, skewing some of the reporting.
• If you're going to do a Level 5 evaluation (Return on Investment) you must do the previous four levels of evaluation first. When data was collected at the end of this case study project, it was determined that not all stores had complied with the training requirements. Had a Level 4 or Level 3 evaluation been conducted (heck, even at Level 1 we would've know if the training was actually going on) this would have been identified much earlier and perhaps the Level 5 evaluation would have been scrapped. Even given the low compliance, this project had a positive outcome; but the outcome is inaccurate due to the non-compliance.
• Identify in advance, what assumptions are being made about the effectiveness or the reach of the training. Again, as in the case of this project, it was assumed that training at the managerial level would trickle down to the hourly associate level. This was an assumption that everyone made, but not one stated out loud. It came as a bit of a surprise to find that no positive outcomes were realized at the associate level. It was also assumed that the stores would conduct the training or send all of their managers out for the necessary training. Compliance was not checked until late in the project, at a point where nothing could be done to remedy the situation and still conduct the evaluation on time.

Happily, this training initiative was a success. Not an overall success, as we've mentioned, but it did have enough positive outcomes to be afforded the label "success." Even if we had failed in the majority of criteria by which success was determined - a number of important lessons would have been learned. It is hoped that you, too, can successful evaluate a training project by heeding the lessons learned in this case study.

Sidebar:

Sidebar:

LESSONS LEARNED

• Specifically identify what you're trying to affect.
• In advance of planning the training - plan for the evaluation.
• Identify well in advance what data will need to be collected in order to conduct the evaluation - and how readily available it is.
• If you want to affect a specific group - apply the training to that group
• No matter what level of evaluation is your goal - you must do the preceding levels of evaluation first.
• To the best of your abilities - identify all of the assumptions you are making about the effects of the training and state them out loud to everyone involved.