Way too many companies think that L&D will put together a plan, and then provide learning to their employees in a quick fashion. They ignore that it takes time. Planning. Strategy. Rollout, and the list goes on. With generative AI, companies are adding it without truly investigating its impact, not just now, but down the road. They bring in the top talent to identify the right LLM or LLMs. They bring in folks with doctorates and consultants who tell them this is the best LLM for your company and your data.
On the learning or training side, you may have the opportunity to sit on a task force or a committee and provide insight into what you need or what you see is a need for the workforce at the company or companies. It seems pretty cut and dry, but when it comes to generative AI and employees, let alone companies making the KING KONG of decisions around adding generative AI, it isn’t.
Scenario – What happens if those on the committee or task force are unaware of what is happening right now, with, say, ChatGPT, even ChatGPT for Enterprise? What if those folks from finance, legal, marketing, sales, and others on those committees are making assumptions based on previous implementations of solutions that, in this case, are false? Or do they read enough to see as though they have it when in reality, they don’t?
When I was at the AI Summit in Amsterdam, the assumption with one talk was that people understand AI, or will enough to make those decisions. What though if they don’t? I have met people who say the consultant recommended this, and in turn they go with it.
What you may ask, does this have to do with my workforce? What does it have to do with L&D or Training?
The people you are providing learning to – Your Workforce
There are two sides to the coin when it comes to generative AI and your employees, i.e. your workforce. For those that are aware of ChatGPT, they are more than likely using it. Directors? Sure. C-Level? Probably a few. Company-wide in general? Sure. Frontline workers and those on the floor? Unsure, although there has been studies out there (FWIW) indicating that blue-collar folks do – it does depend on what the task happens to be.
IF you were to take a poll at your company – which I often wonder how many people in L&D or HR actually do; you will get a better idea of the segments – especially if you make the survey 100% anonymous. Otherwise, you will get people who say they are not using it in work, when in fact they are. Then and only then, will you truly know the actual percentiles.
Let’s though go back to the workforce (sans the survey). One company in the past few months rolled out ChatGPT for everyone to use at the company. Amazing. I do though wonder if they are paying attention to some of the bumps that are showing up? And I do wonder whether the head of L&D there – let’s say CLO in this case, is paying close attention to all the latest around ChatGPT? Because here is just a slight few:
- Prompt leaking is getting worse – This happens with any LLM I should add, but researchers focus on ChatGPT (since it is very popular). The version they eye is 3.5 (ChatGPT), 4 (the paid version, and better than 3.5), Enterprise ChatGPT (the business version, which is mainly like 3.5 Turbo – which vendors in our space tend to go to) and that’s it. 4.5 Turbo available only to developers at this time, is also starting to show issues with something else. The idea that no employee at your company will try to jailbreak ChatGPT, may be foolish on your side. The way to counter it, for all those CLOs out there – is to come up with a gameplan. Ditto for the company themselves – hello Compliance Officer and CTO. L&D will end up getting involved – those who oversee it at companies, who do not have the title of CLO (depends on company, organization, etc.) Ditto again, if you are running training for an internal workforce.
Some zingers that have come as a result of prompt leaking include:
- Getting the directions on how to make illicit items such as napalm (using the grandmother hack)
- Getting personal information, including addresses, phone numbers and other personal data on a random individual (not working at your company) This was achieved multiple times. One way, was the repeating of the word poem, over and over again.
- Getting private company information including the company’s financials. This was the company’s own information, and not from a public generative AI. Rather an enterprise version
Prompt Leaking doesn’t take a Ph.D. to jailbreak, and that is a problem. Anyone can jailbreak, by chance. Granted the ones you hear often are do to researchers testing it or testing samples. Just recently, a publication called Decrypt was able to prompt leak (jailbreak), of Grok, the LLM from Elon Musk which is available only to paid subscribers of X (formally known as Twitter).
While OpenAI will make the fix, the challenge is that this isn’t going to be one-off. There will be more.
For those thinking, no worry I will use another LLM and it won’t happen to me, recognize that any LLM can present prompt leaking. Even with your own data and information; and not public – i.e. the internet.
- Lazy Results – A brand new to start appearing. In November, people started complaining on Reddit that they got far fewer results when using ChatGPT. Many folks scoffed. Well, surprise! It is happening with ChatGPT-4 (the paid version). The AI responds with less characters (words) or says something along the lines that it can’t respond (even though the question is something it could respond to)
A developer saw this using 4.5 Turbo (which was launched in November, available only to developers – as of this post) via an API. By using the words “internal prompt,” they found a less number of characters. They selected a specific date in December. This is relevant, in this case, because the last update to 4 and 4.5 Turbo was in November.
- AI Bias – It is real, it happens when you shove in your own materials, your own reports, data, etc. into an LLM. This is a problem with generative AI. I don’t care if you are using an Enterprise version of an LLM, AI bias can occur
- Hallucinations – One vendor in our space, identifies it as a mistake (when folks are starting to use the generative AI. A mistake is when I am not paying attention to my cell phone, and the President of my country calls me. Hallucinations produce fake and false information. And it doesn’t matter if its your own materials, information or not. I think way too many companies, believe it is their own stuff going into the LLM, then this won’t happen. Sorry, it will.
- I cannot stress this enough: there is no such thing as 100% accuracy. Even vendors who say well it is 98% accurate, can’t guarantee this to take place with your stuff. It might end up 95%, 92%, who knows. The point here is if you are fine with, let’s say 3% inaccuracy and it is happening with your employees using it for their work, and that work happens to be crucial to the company, that will be okay with you? Especially if the employee has no idea that it is wrong? I don’t think so.
The Latest LLMs that vendors are using, and perhaps your company is too
Why is this relevant to L&D and the workforce? You need to know the key pieces around the LLM, not specifics, but enough to train your workforce when they are using this generative AI. Perhaps the LLM is poor at hyperlinks to sites (two LLMs have this issue). Or maybe you can’t place an image into the prompt and have the LLM produce results (many LLMs can’t do this). Perhaps you want to type your question in another language besides English using the more advanced capabilities tied into the LLM (LLMs advanced capabilities are available only in English, although recently an Italian company has been building their LLM in Italian). Perhaps the LLM your company has selected data sets haven’t been updated since 2022. I’d think that is relevant information to know.
As noted in another post and on LinkedIn, the majority of vendors who have generative AI in their system are using ChatGPT 3.5 Turbo. The reason why? Lower token fees. Oh, and they have only one LLM. There are a few that have two LLMs, but one of them is ChatGPT (they might say Azure, which is using the OpenAI ChatGPT 3.5 Turbo LLM). CYPHER LEARNING (their name is in caps) has ChatGPT 3.5 Turbo but also has Stability.ai. Cornerstone? ChatGPT 3.5 Turbo and Llama-2 from Meta.
On the other side, as in what is now out there – here is a short list.
Oh before I forget, in the generative AI space, there is a revolution going on; okay, a strong word there, but when you think of LLMs – you quickly think of a larger model, thus requiring a lot of computing power to go along with all those parameters.
However, an SML changes that narrative. It means a small language model. What is it? Well, it can run on a laptop or mobile device. Now that is small. But before you think that the parameters must be less than a Large Language Model and performance too – HA, you’re mistaken.
Microsoft just launched (as 12-13) Phi-2, which is an SML. It has 2.7 billion parameters. That, BTW is far more than ChatGPT 3.5 Turbo. Microsoft claims it outperforms Gemini Nano (an SML from Google), and Mistral 7-B (seven billion parameters, LLM).
Gemini, which I think can be a game changer (although right now, it is a work in progress), comes in three options. Ultra is for very hard core tasks and reqs – not likely to be used for the masses, and Pro – the more likely usage one.
How many of you are aware of SML right now? I can tell you that for FindAnLMS, the model I will be using will be an SML. Oh, you weren’t aware that I am working on bringing in generative AI for my platform? Well, now you know.
What should I – uh, you overseeing L&D/Training or whatever department that trains your workforce?
I can tell you that what I am finding is a lot of folks are doing nothing or very minimal. I get it. This isn’t just producing a training manual that nobody ever reads on some boring subject tied to the company. Nor is it creating those awful training videos that nobody watches – I mean, they watch them but are thinking, what will they have for lunch?
This isn’t just sending out an email or text or having a group chat/meeting via Zoom and laying out some basic things. Because I can tell you that generative AI is at the flea level, as I noted. We are not even there yet, and what there is going to be is relatively unknown.
For your workforce, though, there are some things to know – and thus you must consider when creating your plan for your workforce through online learning. You heard me right; e-learning is going to be the crucial piece here. Not paper. Not an ILT session. It’s not a vILT session, and not your CEO talking about it, in a recorded video (sorry, but nobody is listening to you).
The STEPS – YOU SHOULD BE DOING NOW – Don’t Wait. Waiting is bad!!!
- Create a course covering the basics around generative AI, specifically about hallucinations, which exist in any LLM (you can ignore that part), so you can just say ChatGPT (or whatever solution you are using). This means fake or false information – and state as that.
- Say that before you place it into a task, or just learn new information (or whatever vernacular you decide on – based on your audience), – the learner needs to review it before sending it off (let’s say the person is creating a deck, and it is a cut and paste thing OR taking that info and applying it to some task they need to do). Correct the information. If they are unsure, ask someone at the company to validate – let them know this is a positive thing to do (I think a lot of people won’t, but if you are using or having mentors, this is a plus for them)
Additional items in the course – do this via a TOC (Table of Contents), or somehow they can go back. I would recommend a video screen recording embedded in the course itself OR another option is the screen recording with a clear voice on what happens when you do this. Trust me, you will find examples of it.
Other items in the course to include
- Do not start with an ambiguous question. The results will be all over the place – plus your company is paying for that LLM, and token fees will escalate when folks are entering in statements or phrases in the prompt. If you are a company with 5,000 or 100,000, those numbers skyrocket, and your company will take a financial hit – especially at the 50,000 or more. This again, is more along them using ChatGPT (for example) for work – and a lot are. This is another screen recording, plus some text. Or just screen recording. I mean you might have a course title and then have specific modules for them. Ensure your vendor provides analytics based on how often the person went to that specific module and how long.
What can ChatGPT do (I won’t cover GPT-4, because it can do far more, stuff, but the token fees even at a quarter of a cent, will increase costs if lots of people are using the prompts, and over and over again, hence try to avoid the ambiguity.
a. Be clear and concise
b. You can do scenarios or ask for examples
c. You can ask it to write the response or text that you place in the prompt in a professional tone or professional style. And you can ask it to respond in a witty tone too.
d. You can contextual the statements or items you are entering into the prompt
e. Refine your responses
f. If they are using ChatGPT at home and not thus using the company’s version or some other generative AI Or using Bard or Bing or another search engine’s LLM (Leo from Brave, for example) – that the images they are shown may be copyrighted. Thus, never use any picture from the sites in any materials they are producing for the company unless they can be sure that it isn’t copyrighted. Ditto on the videos they find.
I have found that when using DALL-E-3 with ChatGPT-4, the images are all over the map. You have to refine and refine, and even then, it is all over the map. Plus, it takes a bit of time, thus, if you expect an image in five seconds, that isn’t happening.
I have seen the same problem with using Bing Create and even MidJourney regarding images. Midjourney is far better than many of them, including Bing Create.
Fun Fact on another Gen AI LLM
I was using Perplexity.ai’s pro version and asked it to create a recipe for chili with Tofu. It presented me with a recipe and showed me videos (something ChatGPT cannot do). They were mainly from YouTube, but there were others out there. While the majority were related to my specific request, I found one for making a dish with Tofu, that had nothing to do with Chili. Relevant to my inquiry? No.
Back to your course info to let your learners know
- You can create a list with bullet points on anything specifically, for example:
Create a list with bullet points on the number of salespeople whose close rate is higher than 45%. Now, the data will be shown – if the vendor is using Enterprise ChatGPT or some other type of Enterprise version because it is the data the company dropped in to train the LLM – it is your company’s own materials, data, etc.
Another example is someone might ask (as an article noted, and I thought it was a great idea) to create a list of skills I will need for my job role (and enter what that is).
You can also create tables and pie charts with ChatGPT (ChatGPT 3.5). If the learner is accessing ChatGPT at home (and not the one the company uses) to extract answers, you need to provide other info. Keep the hallucinations known. They can still do what I showed below, except it won’t be your company’s info.
Plus, the public data sets that ChatGPT was trained on are from 2021. And public of course. Still, always check. There is also the probability of copyright issues – especially with images.
If the end user is using Bing, for example, the information is continuous and fresh – but the hallucinations do exist, Plus ChatGPT’s track record on links going to sites that either do not exist, Or you find is just weird sites, isn’t great.
I strongly recommend going into my Learning Library and selecting the tab, Bard, ChatGPT, and Others, where I post articles comparing each of them, especially Bard (this is Google’s one) and ChatGPT. Bard is adding Gemini Pro into the product, and yes, I do have articles with comparisons using Gemini Pro vs ChatGPT. ChatGPT is in Bing. Extract the information from these, clean it up, thus not using directly unless you will provide citations, and cover it in your course. People should be aware OR at least find some items that are useful – my recommendation, rather than sticking the whole thing into your course.
You cannot stop anyone from using ChatGPT (the freebie version) at home if they are not using a company-issued laptop – where you could do it if you block it. Most people have another computer or laptop or mobile device, which you can go directly to ChatGPT via the web or through an app.
If they are savvy, they may use Poe – which I won’t get into. My guess? ChatGPT is the choice.
If you are using ChatGPT-4 (aka Chat Plus) or they decided to drop $20 a month to do it, and you can – then I recommend the reads, again in my learning library, on what is possible. I will have an article up before the end of the month that covers prompt leaking, the issue with ChatGPT-4 and their plugins, and more. Or you can subscribe to my newsletter, which dives into a bit more – ongoing items and the latest LLMs in an easy-to-read, understandable, and not complex. The more you know, the better you will be able to understand, digest, and apply to your learning/training; whether it is for your employees, members, students, and even customers/partners, etc.
Newsletter on Generative AI
Quick Items
Back to the course (or courses). You can do this in short (micro-learning) or go long. Duration, BTW is irrelevant, but I know people want to know. What takes me five minutes may take someone 30, plus you want them to go back into this, and not just a one-time.
Anyway:
a. Look at what you are planning to do, and see if you can apply or use it with the LLM you are using – my guess, is over 90% is a yes. But, always, always verify. A human is a must.
b. If you have a leadership and development program, you can use generative AI for it – but check the results.
Always remember to be specific. Crucial. I cannot stress this enough – which is why you must tell your employees/learners.
You may want to let your employees know about AI bias, or you can keep it to yourself (my recommendation). It happens in AI, and I don’t care if it is your own materials – thus private – going into the LLM. It does happen.
For yourself, be aware that even with your own private information, data, content, etc. an LLM, is not 100% accurate. Never will be. Some folks say 98% or 95% accurate, but that is up for debate. Plus, let’s say Junior is creating a report that will impact the company’s strategy plans for 2024. And Junior gets information that isn’t accurate. Junior doesn’t know this. Junior puts that information into the report. Your company doesn’t do verification. What happens next? Is it worth 5% inaccuracy? I would give this as an example in your course – but don’t use Junior as the name. Repunzal or Ebenezer sounds better. Plus, what is the probability someone at your company has either of those names?
Privacy is a huge issue, especially with any system that uses generative AI – but in this discussion, let’s focus on vendors in our space or learning tech that use generative AI.
a. There are options – you can select a vendor that offers a privacy layer over the LLM that goes beyond what is included. For example, one company can strip out 54 Piis. You can find an LLM that uses RAG (you can search to find out).
b. The systems that are using an LLM from say OpenAI, Azure, A21, Anthropic, Google Cloud, AWS and other commercial ones, have privacy components in them, which is sufficient. If you want to extract Pii, then you need to ask that specific vendor – i.e. ask your learning system or learning tech vendor, which may or may not know – and then they need to reach out to the provider. Llama-2 from Meta has privacy capabilities in it. To date, I have yet to find a vendor that isn’t using one of the bigger names out there. The ones I have found so far are using an LLM from OpenAI, Azure -which is Open AI, Anthropic – Stability.ai, Google Cloud’s model as a service, with the base being Vertex AI – machine learning, Llama-2, and Bedrock a model as a service from AWS (and OpenAI within it). And to date, I have found just a few vendors that are using multiple LLMs (a good way to go)
c. Copyright of images is an issue – however, I found one vendor that has an agreement in place with a company that provides images, removing the copyright problem.
Other items that I have seen that are concerning to me, with vendors who have jumped into generative ai (in 2023). I would not recommended doing so, unless you knew all the ins and outs with generative AI. Which based on what I have seen indicates that, some are doing it, but more are not.
- They do not know all the regulations that countries or the EU have implemented for AI. They are not all the same,; they must ensure their system is compliant. California has its regulations in the states – similar to the EU, but not exactly; New York has their own, and other states are going that way (clearly not all). India has their own, the US has started (but I am not expecting EU level), China has its own, Australia has its own, and other countries I did not mention may have their own. Only one vendor I am aware of has a 3rd party company keeping track of all the regs and passing it through to the vendor. These regulations will continue to change; a vendor must know this and understand and implement it. And it will take more than one person, IMO to deal with this.
- Some of the responses around not listing information in their system about hallucinations that might occur are frightening to me. I have seen this with a few vendors. Holy Moly. And their reasons for doing it, are concerning. Again, not just one vendor, multiple vendors. This isn’t a game. This is a must. If you are creating a course (which is popular with vendors for generative ai) you need to let folks know this. Ditto, say AI bias. I have seen only two vendors really stipulate this, with one, on everything that generative ai does in their system – which is the way to go. The other vendor, which has generative ai on the learner side, mentions it, but in small text.
- None of them have context windows (the prompt window you see) that, thru icons or whatever, allows the admin or whomever to identify that if the information is incorrect, what is the correct response and submit it. Despite them seeing this in any LLM out there, such as ChatGPT-4. The AI learns from itself, so if you edit the response, which even the vendor does, and save it, the generative AI doesn’t know this. Thus wrong response, the AI says, okay, it must be right.
- Pricing for those tokens, the cost to you, is really low right now – because first the cost per character is low – but the systems that have gen ai, only a few have it on the learner side, so yeah, it should be low cost to the client. A couple are rolling it out on the learner side next quarter. What I have seen are three pricing angles:
a. It is included at no additional charge. The vendor eats the cost. This may change once it goes onto the learner side, and you get a lot of people asking questions, or statements, or whatever into the context window.
b. The client purchases a bag of tokens – i.e. you pay for a package of tokens. One vendor has something like 20,000 tokens in a package, and the price point is low. However, they haven’t hit the learner side yet.
c. The generative AI in, the content creator or assessment tool, etc., is an add-on. Thus an additional cost.
Bottom Line
Your workforce and generative AI, need guardrails from the people running L&D and Training, or whatever department produces content for their learning system. These are guardrails to appear in a course or content, not just the LLM that your company trained its data on (btw guardrails do not eliminate hallucinations).
Generative AI can be a wonderful plus to any company and, thus, any system. But we are at a flea stage, and it will only improve and get better.
There are signs indicating that AI bias is showing up with minorities.
I am surprised how many vendors are not paying attention to the entire market, LLMs or issues around it, and so on. This isn’t just stick a plug-in that has generative AI here or let’s add it, and we are going to see what else we can do.
This is the Industrial Revolution 2.0 – a massive game changer globally, and in our case, in any learning system or learning technology. Your company will likely add generative ai (if they haven’t already) by the end of 2024 and definitely by the end of 2025.
Thus your workforce will use it. You can’t block them though from using whatever they want at home. And therein lies a real problem. Because they could extract it and use it in their workplace.
As the COO of OpenAI noted – a company should not expect their productivity to be solved with generative ai alone. It is nowhere near where it will go.
You must have a human element here.
Not just on the workforce side,
But on your side.
E-Learning 24/7
