The Problem with Edtech Review Services

Education Week published an article on edtech review services in August 2013 that concluded with the following statement:

As the review sites evolve, observers are watching to see which will emerge as the “go to” resources for educators making decisions about technology tools.

“What they’re trying to do is a noble goal, because I do believe that teachers have an overwhelming number of choices in the marketplace,” said Rob Mancabelli, the co-founder and CEO of BrightBytes Inc., a San Francisco-based learning-analytics company.

“One of the things the review sites need to do better is to base the reviews on data,” he said—for instance, matching a student’s areas of deficiency with technology proven to address those shortcomings.

Mr. Chatterji, who is working on EduStar, agrees with the need for more data.

“We’re all chasing after the same North Star, which is to raise educational outcomes with technology,” he said. “In the ecosystem of [review-site] players, the gold standard is evidence.”

I totally agree. How is an educator or parent to know which math app they should purchase? Which will definitively lead to an advancement of their student’s or child’s knowledge?

After thinking obsessing over this problem non-stop for a few years, I’ve come to the conclusion that it is extremely difficult, if not impossible, to get this kind of information.

Why?

  1. Which metrics should you use?

    The student’s scores in the app? Not all app vendors will be willing or able to provide this data. Even if you have it, normalizing it for comparison against thousands of other apps is a monumental feat. How about the student’s standardized test scores? Those metrics are already questionable, as any educator will tell you.

  2. How do you conduct a scientifically-controlled evaluation?

    There are many factors that influence a student’s education, from teachers to parents to peers to instructional materials to socio-economic factors to genetics to, well, you get the point. Determining whether a specific app has made a discernible difference is a significant challenge. Some vendors already try this, but knowing how an app will perform in a classroom or home full of distractions versus a pristine laboratory are very different things.

  3. How do you account for individual differences?

    Some students will take to a game and learn a lot from it, others will just learn how to cheat the game. Some will excel with just a bit of hands-on guidance, others will need constant attention. How an app performs depends a lot on the learner. Does this mean the app is effective or ineffective? Unfortunately, one size does not fit all.

  4. How do you account for rapidly changing products?

    Let’s say you do find a way to conduct a scientifically controlled test in a classroom environment using a reliable metric. The likelihood that the developer will read your report and incorporate your findings is very high. Apps are constantly being improved and evolved. What does that mean to your research? It is out-of-date the moment a new version is released.

There are some fantastic and noble efforts underway to solve this challenge, though I think we are all a long ways off. That gold standard of data and evidence is monumentally difficult to get in a repeatable, reliable, personalized, and scalable way.

One solution involves using rubrics through which to evaluate all of these apps. Though the evaluation requires manual work from someone trained in pedagogical assessments, it is a reasonable proxy of quality, if not true efficacy. Many current edtech review services use some kind of rubric, with some differences in breadth and depth.

Another involves in-depth reviews written by an individual with a good understanding of the domain of the app. This is similar to the product reviews seen on popular electronics guides. Such reviews are limited to the expertise and biases of the reviewer, though they can at least provide information from someone who has used, poked, and prodded the app.

One of the weaker solutions is the use of ratings. Since they are relatively easy to get from reviewers, they harness the collective intelligence of the crowd. Ratings are also easy to understand at a glance and to use as a basic filter. However, they fail to provide any context. Some services get around this by offering multiple rating dimensions, but these can also suffer from the problem of averaging out very positive and very negative ratings.

These solutions sit on a spectrum with the wisdom of the crowds on one end and the wisdom of experts on the other.

If you believe the pedagogical value of an educational app could be obtained from the crowd, then try the Yelp model. If you believe the pedagogical value needs to be evaluated by experts, then try the Consumer Reports model. Edtech review services are currently employing strategies all over that spectrum because getting to a true gold standard with actual data and evidence is so difficult.

Will these solutions be enough? Will someone be able to crack the gold standard? I’m not sure it is possible, but I sincerely hope someone will.

Edtech Trend: Content Creation

The impact Khan Academy has made on the edtech market is widespread. One such result is in the growing market of content creators.

Though there has always educational videos online, the simple bite-sized format of Salman Khan’s videos attracted a wide audience on YouTube. Then donations from prominent investors like Bill Gates propelled his humble efforts into a full-fledged non-profit.

With this attention came criticism from educators. Some reacted to the grandiose statements made about him in the press. Others reacted to the pedagogical content of Khan’s videos. And others decided to create alternatives. At one point, I was able to count 60 sites similar to Khan Academy. There may be more now.

At the same time, a handful of entrepreneurial thinkers realized they could make it easy for anyone to create Khan Academy-style (i.e. digital whiteboard) videos. Instead of using someone else’s videos, you could create your own that are tailored for your students and curriculum.

This is just one example of the educational content creation trend. Right now, I see tools creating the following types of content:

  • Videos – Digital whiteboards, lectures recordings, screencasts, etc.
  • Audio – Podcasts, songs, etc.
  • Images – Lesson plan illustrations, diagrams, infographics, etc.
  • Animation – 2D cartoons, 3D cartoons, etc.
  • Presentations – Lecture slides, project presentations, etc.
  • Lesson Plans – Online multimedia lesson plans, offline lesson plans, etc.
  • Books – Textbooks, ebooks, storybooks, etc.
  • Interactives – Math manipulatives, simulations, maps, timelines, etc.
  • Websites – Class websites, blogs, wikis, etc.
  • Games – Mobile games, desktop games, web-based games, etc.
  • Quizzes – Exercises, worksheets, polls, etc.
  • Video Quizzes – Quiz questions mapped to certain points of a video
  • Online Courses – Instructional media followed by assessments
  • Digital Stories – Animated multimedia stories
  • Portfolios – Examples of student work
  • Electronics – Robots, mechanical devices, pre-made kits, etc.

One of the newest types of content to the market are online video quizzes. I am sure there will be many more to come.

This trend is part of the larger maker movement that many teachers and students are wholeheartedly embracing. It empowers teachers to craft materials suited for their classrooms and students to demonstrate their learnings and creativity in an engaging way.

Content creation going into the hands of teachers and students is a significant shift for education. Though the largest publishers may continue to dominate the market for some time, content created from the bottom-up (teachers and students) will increasingly augment content from the top-down (publishers). Bottom-up content won’t replace top-down content entirely, as there will always be a need for standardized materials, but the growing number of new publishers with high-quality, low-cost content will certainly be a threat the big publishers. And many of these new publishers are working with or are a part of the content creation trend.

There is already a wide diversity of offerings. This number will continue to increase, as well the variety of content types available. This will mean a wider range of quality in the content too. That’s where aggregation and curation will be necessary, so the most relevant and highest quality services and content can be surfaced.

What do you think?

Edtech Trend: Aggregation and Curation

The market for education technology is exploding right now. As of this article, there are over 90,000 iOS apps in the education category of the iTunes App Store, and over 90,000 Android apps in the education category of the Google Play marketplace. That’s 180,000 mobile apps between iOS and Android. If you throw in the hundreds of thousands of educational websites and desktop apps as well, that is a lot of edtech through which to wade.

This explosion is fueled by:

  1. A growing group of entrepreneurs passionate about education, as well as tech-savvy, entrepreneurial teacherpreneurs
  2. Advances in technology that have made the development of websites, apps, and digital content easier and cheaper
  3. The introduction of Internet-capable mobile devices and apps
  4. Increasing investment capital from investors

But with all of this technology available, how is a busy educator to sort through this ocean of resources? Where would one start?

The answer is in aggregation and curation services:

  • Aggregation services gather all of the resources available into a single destination. Some aggregators also label and organize them into a searchable directory.
  • Curation services sort and filter the resources so the worthwhile ones rise to the top. This can be done using computer algorithms, human selection, or a mix of the two.

Such services are already emerging. There is a number of aggregation and curation services for educational videos, lesson plans, online courses, mobile apps, etc. on the market, and that number is growing. I predict that this trend of aggregation and curation services is going to continue.

The diversity of such services will also increase. There are many ways to handle curation, as mentioned above (i.e. computer algorithms vs human selection). While each has its pros and cons, we will see services experimenting throughout that range. I suspect the right balance will fall somewhere in the middle.

Personalization will also be a factor. To curate for a particular classroom or student, the resources selected must be personalized to each specific context, because in education, one size does not fit all. This requires some awareness of the students involved. Current trends in data privacy will have an influence into how this awareness is collected, but without a personalized service, you run the risk of “solving for the middle,” which is just as bad as “teaching to the middle.”

We will also see these services as features of existing products. Within educational content creation services, curation will be an important factor in surfacing quality content.

One last prediction: This trend of aggregation and curation services will put many new solutions on the market before they consolidate and the least effective services fall away, just like the resources they will be aggregating and curating.

What do you think?

How Can An Edtech Startup Enter the K-12 Market?

The answer is the same for any other market: Find an agonizing problem that a significant number of people have, then offer a better solution than the existing alternatives.

And I would add: Identify a sustainable business model for your solution as well. It does schools no good if you build a great solution, only to make no money, run out of savings, get no funding, and close up shop. I know this from painful personal experience.

It may seem difficult to make money in the K-12 market, but there is more money than most realize. If your customer is a K-12 school district, their total IT expenditure in 2011 was $9.4B.

Also, all school districts want to know that you have a sustainable business model because they want to know that you will be around for years and years, rather than be a transient startup that could disappear at any time. This is a very important point that many aspiring edtech entrepreneurs don’t realize. Selling to schools is a B2B business model. Both businesses and schools prefer to procure products from companies that will be around – and schools even moreso, because integration can be such an intensive process.

This is in stark contrast to individual classroom teachers who don’t control a budget, however. Selling to teachers is a B2C business model where most consumers have come to expect free products.

My suggestions for an aspiring edtech startup entering the K-12 market are:

  1. Validate that your product is solving an agonizing problem.

    This problem should make your solution a “must-have” and not a “nice-to-have.” The average teacher does not have a lot of extra mindshare for new products that are only a nice-to-have. They have to need it. A lot can be written about this. If you have not done so already, go read The Lean Startup by Eric Ries, since his methodology can apply well to B2B businesses.

  2. Validate that your solution solves that problem effectively.

    Find some local teachers or students and watch them use your product. Test the outcomes using the scientific method. See if there is a demonstrable and repeatable improvement from the use of your product. Finding teachers or students willing to do this is not an easy task, but if you are truly solving an agonizing problem, it shouldn’t be that difficult. Being able to come up with quantifiable data to prove your solution will help you win more customers and provide you with a valuable case study and testimonials. These teachers or students are essentially your innovators, as detailed in Geoffrey Moore’s book Crossing the Chasm.

  3. Identify your actual buyers.

    In the K-12 space, the end-user is often not the same person as the buyer. There are many buyers in a school district. They can range from superintendents to principals to IT directors to curriculum directors to department heads. Some end-users can at least be influential in the buying process than others. If you cannot identify the buyer, your end-users may be able to help you too. Note that schools often pay with purchase orders (POs), which could take the form of a check that will arrive in months.

  4. Validate that your buyers will pay for your product.

    Concentrate on making that first sale. Your price point doesn’t matter that much in the beginning. As long as it is not absurdly low, don’t obsess over the right price. If you are lucky, you will have a buyer who is willing to work with you on a reasonable price. Look at competitive products and their pricing structures as a basis for yours. Understand the issues that the buyer cares about. Having a case study to validate your efficacy will go a long way in convincing them. Some may even require this kind of proof first.

Once you have made that first sale, listen to your end-users and buyers, then tweak your product as necessary. Meanwhile, go focus on your next 10 sales. Then your next 100. Then 1000. Going in small incremental steps will keep you realistic.

Or even better, look into different distribution and viral growth models that can accelerate your sales. The best kind of growth is the kind where your customers naturally want to tell their colleagues about you.

This is a simplified view and each step has many nuances, depending on the exact nature of your solution. Hopefully this provides enough of a starting point for any aspiring edtech entrepreneur.

This article was originally posted as an answer on Quora and modified for this blog.

How to Be a Product Manager with a Portfolio of Thoughts

I spoke with someone today about being a product manager. He is completing his MBA soon and has little work experience. “Without much experience, how can I get a job as a product manager?” he asked.

Here is what I told him.

Companies want to know if you have the right mindset, critical thinking skills, and culture fit. If you don’t have the work experience, you can do this with what I call a “Portfolio of Thoughts.”

Just as designers have portfolios and developers have code samples, product managers can demonstrate their skills with a collection of essays on how they think through the development of a product. This can be as simple as a blog.

For each entry, I would recommend topics such as:

  • Picking a particular problem and proposing a solution for it, in as much detail as possible. Go through your thought process and rationale.
  • Picking an existing product and offering a balanced critique. If you love this product, why? And if you don’t, why?
  • Picking an existing product and proposing how you would make it better. Again, offer your rationale.

Those three topics ought to provide you with enough material to accumulate a portfolio of thoughts. A recruiter who reads this will get a great sense of your product sensibilities and thought process.

After you have a long list of posts, don’t worry about going back and revising old entries. Feel free to add updates though. Include a note indicating that you have updated the post too. Everyone’s skills grow and evolve over time. What you have written about a product in the past may differ from what you would write now, so an archive of your thoughts can serve as a demonstration of how you have grown as a product thinker.

Also, take the time to follow and read the blogs of other product managers. Read books on being a good product manager. Meet and have coffee with experienced product managers. Pick their brains. Study good products. Study bad products. Study this craft. And keep up your portfolio of thoughts.

In lieu of a job, doing all of this will grow your skills while providing recruiters with a fantastic way of getting to know you.

As an added bonus: This will be a great way to build your personal brand too.

How to Find Developers for Your Edtech Startup

Are you an educator with a great idea? Need a developer to help you build your product?

Note: This article is written for educators who are interested in creating a startup, though aspiring founders from other fields may find it useful as well.

You are unfortunately part of a large sea of people with great ideas. The startup world is littered with more idea people than implementation people. Being an educator, you probably aren’t exposed to many developers. But don’t worry, there are still ways you can find developers.

Look through your own network
Start with people you know. You may already have a developer or two as a 2nd or 3rd degree connection. The closer the connection, the better. This person doesn’t need to be an experienced developer with a Computer Science degree; someone who is willing to learn can be just as good. Building out your idea could be a great way for them to grow their skills.
Partner with technical students
If you are a K-12 educator, your school may have a computer class or workshop with budding technical geniuses. Some of the best developers started out programming at a young age. The chance to make a web or mobile app could be very enticing. However, be careful not to take advantage of a young student’s eagerness. As an authority figure, they will listen to you, perhaps even at the detriment of their homework and studies. Some may need some technical guidance as well. Though it’s not easy, try to find an edtech developer who is willing to mentor your student. If you are in higher education, your university may have programming classes. College students make better partners than high school students, since they have more life experience, need relatively less guidance, and may even genuinely be interested in starting a company with you. Their project with you could be bundled as a class assignment as well, and they have the added benefit of potentially bringing other students into the team.
Work for an edtech company
This isn’t a viable option for everyone. A colleague with whom you’ve worked is always better than a smart stranger. The only way to have technical colleagues is by having worked at a company with them. This method could mean a long journey however, as you can’t just work at an edtech company, than leave in a few months and expect to have formed deep relationships with their developers. (Not to mention potential bridge burning and non-compete clauses that some states carry.) It’s a small industry and you could lose a lot of goodwill that way. It may take at least a year at an edtech company to create meaningful relationships there.
Go to a Startup Weekend EDU
Startup Weekend EDU is a weekend-long “hackathon,” where various people come together to build a website or mobile app. It is an intense adrenalin- and caffeine-fueled event. The audience includes idea people, marketers, designers, and developers. The first day starts off with all the idea people pitching their ideas to the whole room. Then everyone is free to walk around to learn more about each idea. The best ideas eventually attract teams – which hopefully includes developers too. It’s not easy to attract developers, but it is possible if you know how to appeal to them.
Go to edtech meetups
Meetups are informal gatherings of people facilitated by the site Meetup.com. Some cities, though not many, have a large enough edtech community to host edtech-specific meetups. Silicon Valley has one, but not many others do. These meetups tend to attract more entrepreneurs than educators, including some developers. Ed-Tech Meetup is one of the more popular ones in San Francisco.
Be an inspirational champion for an cause
Build up your personal brand both offline and online. Get involved in relevant organizations and volunteer groups. Become a recognized leader. I know of one charismatic individual who has done this on Quora, Twitter, and guest blog posts. He doesn’t have a technical background, but his passion and charisma is very apparent. This isn’t a guarantee of finding developers, but it can make it easier to convince one to join you.

These tips can also help you find a technical cofounder, though knowing whether or not someone will make a good cofounder is an entirely different topic. At the very least, these can help you find developers to help build an MVP (Minimal Viable Product) and get your edtech startup off the ground.

This article was originally posted on the Edtech Handbook and modified for this blog.

The Pros and Cons of Industry-Specific Startup Incubators

Startup incubators (or accelerators) have gotten very popular lately because they provide a lot of benefits to startup founders, from investor signaling to founder mentoring to fundraising opportunities. Their demand has risen so high that industry-specific incubators have appeared.

This is a natural evolution of the startup incubator model. Since each industry has its own dynamics, founders of incubators probably realized they can accelerate startups that much quicker if they provide industry-specific benefits.

But is this true? Are industry-specific incubators worth it?

On a high level, only a few incubators can really make a significant impact for a startup, whether they are general or industry-specific. If you are comparing these types of incubators for your venture, here are some pros and cons to consider:

Horizontal (general) incubators have:

PROS
  • Access to a wider network of investors
  • Access to a wider network of founders in their alumni network
  • A higher chance of getting you press in general tech publications
CONS
  • Little to no industry-specific guidance
  • Little to no industry-specific connections
  • A lower chance of exposing you to targeted and relevant investors
  • A lower chance of getting you press in industry-specific publications
  • A lower chance of industry-knowledgeable founders in their alumni network

Vertical (industry-specific) incubators have:

PROS
  • Access to targeted and relevant investors
  • Access to industry-specific guidance
  • Access to industry-specific connections
  • A higher chance of getting you press in industry-specific publications
  • Access to industry-knowledgeable founders in their alumni network
  • CONS
    • A smaller network of investors
    • A smaller network of founders in their alumni network
    • A lower chance of getting press in general tech publications

    I’m sure you can think of some exceptions to these points. Some vertical incubators are great at getting general press, for instance. The right choice for you depends on your needs, particular industry, and the strength of the industry-specific incubators open to you.

    It is also important to note that some incubators don’t mind if you participate in multiple incubators. There have been a few startups that have participated in both a general and an industry-specific incubator.

    Are there any other pros and cons I’ve missed? What do you think?

    How to Find K-12 Teachers for Edtech Product Feedback

    “Every company’s greatest assets are its customers, because without customers there is no company.”
    - Michael LeBoeuf

    If you are building an edtech product aimed at K-12 teachers, you’re going to need feedback from them. Be it for customer discovery and validation, market research, beta testing, or something else, it is critical to talk to your customers.

    You may be surprised to hear that there are many enterprising educators out there. I don’t mean they are profit-seeking; I mean they are inventive, progressive, and oftentimes tech-savvy. Some call them edupreneurs, some call them teacherpreneurs. Whatever you call them, as a startup, you can see them as innovators or early adopters, in “Crossing the Chasm” parlance.

    This means they are willing to try out untested software. While they are not representative of the mass market, they can be a good reference customer for the majority later.

    So how do you go about finding K-12 teachers who are willing to help you out? The resources here can help you identify and connect with teachers directly.

    Use Personal Connections

    Chances are, there are people within your own network that can connect you with teachers.

    Your own teachers
    If you’ve been through the K-12 education system, reach out to your former teachers. Since they know you, they may be wiling to help. Keep in mind that they may not be early adopters though.
    Your children’s teachers
    If you are a parent with children in the K-12 education system, go to their teachers. Since they know your children, they may be willing to help, and are also likely to be part of the mass market too.
    Friends and family
    Your own network of friends and family may contain, or be acquainted with K-12 teachers. 2nd and 3rd degree connections may not be as willing to help, but will likely be part of the mass market as well.

    Use Educator to Entrepreneur Networks

    As the edtech revolution is growing, so are the resources available to edtech entrepreneurs. Some organizations have risen up to complete the missing link between educators and entrepreneurs, though some are not as active as others.

    edSurge Tech for Schools Summits
    These are a series of local conferences for educators and entrepreneurs. For a fee, you can present demos, offer trials, and solicit feedback. Teachers can evaluate products and suggest implementation plans for their schools. This is more suited for beta and final products than for customer discovery.
    Teacher Tech Talk
    This is an exciting new effort to physically bring entrepreneurs and educators together. Discussions are focused around the concerns of educators rather than entrepreneurs, making this a great avenue for customer discovery.
    edUpgrade
    This is a non-profit organization that connections beta products with educators willing to try them out. Products must be in an experimental state and companies must be willing to work with educators on a regular basis. This is not a good avenue for marketing or customer discovery.

    Use Online Teacher Personal Learning Networks

    A lot of teachers seek to form PLNs as a way to informally increase their professional development. The resources and guides that help teachers do this can be reverse-engineered to find tech-savvy innovators and early adopters. Here are some of the more popular resources.

    Edmodo Communities
    Since Edmodo is a social network specifically for teachers, this is a great place to find them. Like any other community, don’t jump right in and start promoting your products, however. Understand the etiquette of the community before making any posts. Or identify specific teachers and contact them directly.
    Ning networks for teachers
    Ning is a social network platform that allows anyone to create a social network like Facebook or Edmodo. There are many teacher-specific networks out there, though Ning doesn’t include a directory of them all. Some are more active than others.
    Twitter
    A lot of tech-savvy teachers use Twitter as a means to expand their PLNs. It’s a relatively low-effort way to publish and consume information. Most prefer it over Facebook too. You can often find teachers using hashtags such as #edchat#edtech#education, and others specific to conferences, organizations, and subjects of which they are a part.
    Tumblr
    There is a vibrant and engaged community of teachers on Tumblr. Their demographic tends to be younger and more tech-savvy. Most tend to use the tags #education#edtech, and #teaching. You can work backwards from these tags to find teachers. Most have some kind of contact form on their profile pages.
    Pinterest
    According to recent comScore statistics, Pinterest’s demographics skew heavily towards female. Most are in the 25-34 age range and 50% have children. Within this audience is an active community of teachers who’ve been using Pinterest to pin project ideas, lesson plan ideas, and other educational resources they can use. Though you can’t send a message to teachers directly from Pinterest, some include a link to their other social media accounts.
    Diigo Groups
    Diigo is a popular bookmarking tool amongst tech-savvy teachers. They use it to organize all the websites they use. Many users don’t fill out their profiles, so it’s not always easy to reach out to teachers from this source. But some do include links to their other social media accounts.
    Teacher forums
    There are dozens upon dozens of online forums for K-12 teachers, though quality varies significantly. Some, like A to Z Teacher StuffThe Teacher’s Corner, and WeAreTeachers have fairly active communities. That also means they moderate their posts and don’t tolerate product promotion posts. Lurk in the community to understand the members and etiquette before joining in. You can also find forums on specific subjects if you search enough.
    LinkedIn Groups
    There aren’t as many active teachers on LinkedIn as on other social media sources. However, you’ll be able to find a handful of active groups of teachers with some effort. LinkedIn has the added benefit of providing a way to connect with the teachers you find.
    DonorsChoose
    Here’s a little-known tip that may not be suitable for every startup. If you see a project in which your product could be used, help that teacher fund the project, then send the teacher a free copy of your product. Teachers love free stuff, and if your product solves the problem well, you may have earned a new product evangelist.

    Use Face-to-Face Sources

    Sometimes meeting with a teacher face-to-face is better than an online interaction. You’ll find them more forthcoming with information, especially about controversial topics.

    Craigslist
    A fair number of teachers peruse the education jobs section, usually for part-time work. Though it costs money to post a listing in the jobs section, a well-timed listing (i.e. during Winter break, Spring break, a long weekend, etc) may attract a lot of responses. For best results, offer some kind of compensation, such as a free meal (for a face-to-face meeting) or gift card.
    Teacher conferences
    There are hundreds of conferences for teachers across the country. Most won’t be relevant to you. Identify a few that are worth attending and focus on those, such as ISTE (International Society for Technology in Education), one of the largest for IT professionals in education.
    Meetups
    Unfortunately, not all cities have a large enough edtech community to host meaningful edtech meetups. Silicon Valley has one, but not many others do. These meetups tend to attract more entrepreneurs than educators. But popular ones, such as Ed-Tech Meetup in San Francisco, do host events with educators on occasion.

    Innovator and early adopter K-12 teachers are out there, many of whom would be eager to meet you. It’s just a matter of doing the legwork to find and connect with them.

    Know of other resources? Please leave a reply and let me know!

    This article was originally posted on the Edtech Handbook and has been updated for this blog.