UE 5.5.4 Project with LLM Integration
Worldwide
I need to build a UE 5.5.4 project that utilizes either: 1- A local LLM (Gemma 4 E2B or Gemma 4 E4B) prompting it in a multi-model fashion 2- An online service (OpenAI's GPT models, Anthropic's Claude, Google's Gemini, xAI's Grok, DeepSeek) I can switch between Local and Online as a user and for each mode I can choose the needed model. The project has one scene with (Ultra Dynamic Sky) and a single slider widget that control the time of day and a selector to choose between its existing weather presets (clear, rainy, foggy, snowy, etc...). This manual part is already done. I need to add a prompt box at the bottom of the screen that allows me to prompt the LLM (Local or Online) with multi-modal input: - Text (In any language) - Image file (Picked from File Browser) - Viewport screenshot of the current 3D scene (without the UI) - Audio file (Picked from File Browser) - Mic input (Spoken in any language) and the LLM will respond with: - Translate the text from whatever language it was to English (for internal processing) - Changing the time of day slider and weather selector to the requested value (Example: If I tell it set the time to sunset or set it to 18:34 or set the weather to be cloudy, it will do it in both cases) - Transcribe the audio file to text (and then translate that to English text) - Analyze and describe an image (either the image file or the viewport capture). For example if the image is of a rainy environment or of a sunset (or both) then I can tell it "Match the time of day and weather scene in the image" and the LLM will execute the function to change the time of day or weather. - If the user asked a question, it will answer based on a documentation website (GitBook) so its knowledge comes from that specific website. Effectively acting as a Help/Support Bot. The URL of the text file to be fed to either the local or remote LLM for knowledge is (https://creadev.gitbook.io/creadev-docs/llms-full.txt) Note: - If Local LLM mode is selected, I need to be able to switch between Gemma 4 E2B and Gemma 4 E4B on the UI so I can experiment with both. - if Online LLM mode is selected, I need to be able to switch between (OpenAI's GPT models, Anthropic's Claude, Google's Gemini, xAI's Grok, DeepSeek) and provide an API key for each of them. - All of this should be working in a standalone shipping build on Windows, Mac, Linux, iOS, Android. - There should NOT be a requirement to have a separate Ollama Server running, it should be fully self contained (the .gguf file is embedded in the executable) - Must work with Unreal Engine 5.5.4 - Use Blueprint, not C++ - Please submit your estimated time and cost with your proposal.
- Less than 30 hrs/weekHourly
- < 1 monthDuration
- IntermediateExperience Level
$15.00
Hourly- Remote Job
- One-time projectProject Type
Skills and Expertise
Activity on this job
- Proposals:10 to 15
- Last viewed by client:6 days ago
- Interviewing:2
- Invites sent:4
- Unanswered invites:2
About the client
- Canada1:25 AM
- Individual client
Explore similar jobs on Upwork
How it works
Create your free profileHighlight your skills and experience, show your portfolio, and set your ideal pay rate.
Work the way you wantApply for jobs, create easy-to-by projects, or access exclusive opportunities that come to you.
Get paid securelyFrom contract to payment, we help you work safely and get paid securely.
About Upwork
- 4.9/5(Average rating of clients by professionals)
- G2 2021#1 freelance platform
- 49,000+Signed contract every week
- $2.3BFreelancers earned on Upwork in 2020
Find the best freelance jobs
Growing your career is as easy as creating a free profile and finding work like this that fits your skills.
Trusted by