UE 5.5.4 Project with LLM Integration

Posted 6 days ago

Worldwide

Summary

I need to build a UE 5.5.4 project that utilizes either: 1- A local LLM (Gemma 4 E2B or Gemma 4 E4B) prompting it in a multi-model fashion 2- An online service (OpenAI's GPT models, Anthropic's Claude, Google's Gemini, xAI's Grok, DeepSeek) I can switch between Local and Online as a user and for each mode I can choose the needed model. The project has one scene with (Ultra Dynamic Sky) and a single slider widget that control the time of day and a selector to choose between its existing weather presets (clear, rainy, foggy, snowy, etc...). This manual part is already done. I need to add a prompt box at the bottom of the screen that allows me to prompt the LLM (Local or Online) with multi-modal input: - Text (In any language) - Image file (Picked from File Browser) - Viewport screenshot of the current 3D scene (without the UI) - Audio file (Picked from File Browser) - Mic input (Spoken in any language) and the LLM will respond with: - Translate the text from whatever language it was to English (for internal processing) - Changing the time of day slider and weather selector to the requested value (Example: If I tell it set the time to sunset or set it to 18:34 or set the weather to be cloudy, it will do it in both cases) - Transcribe the audio file to text (and then translate that to English text) - Analyze and describe an image (either the image file or the viewport capture). For example if the image is of a rainy environment or of a sunset (or both) then I can tell it "Match the time of day and weather scene in the image" and the LLM will execute the function to change the time of day or weather. - If the user asked a question, it will answer based on a documentation website (GitBook) so its knowledge comes from that specific website. Effectively acting as a Help/Support Bot. The URL of the text file to be fed to either the local or remote LLM for knowledge is (https://creadev.gitbook.io/creadev-docs/llms-full.txt) Note: - If Local LLM mode is selected, I need to be able to switch between Gemma 4 E2B and Gemma 4 E4B on the UI so I can experiment with both. - if Online LLM mode is selected, I need to be able to switch between (OpenAI's GPT models, Anthropic's Claude, Google's Gemini, xAI's Grok, DeepSeek) and provide an API key for each of them. - All of this should be working in a standalone shipping build on Windows, Mac, Linux, iOS, Android. - There should NOT be a requirement to have a separate Ollama Server running, it should be fully self contained (the .gguf file is embedded in the executable) - Must work with Unreal Engine 5.5.4 - Use Blueprint, not C++ - Please submit your estimated time and cost with your proposal.

  • Less than 30 hrs/week
    Hourly
  • < 1 month
    Duration
  • Intermediate
    Experience Level
  • $15.00

    Hourly
  • Remote Job
  • One-time project
    Project Type
Skills and Expertise
Mandatory skills
Unreal Engine
Blueprints
Activity on this job
  • Proposals:10 to 15
  • Last viewed by client:6 days ago
  • Interviewing:
    2
  • Invites sent:
    4
  • Unanswered invites:
    2
About the client
Member since Oct 19, 2024
  • Canada
    1:25 AM
  • Individual client

Explore similar jobs on Upwork

Framing Simulator 2025 Map DevelopmentFixed-price‐ Posted 3 weeks ago
JavaScript
GIS
Android
Java
C++
Gaming Console
Adventure
Action
Ages 7+
Unity
Game Development
QA Management
Game Design

How it works

  • Post a job icon
    Create your free profile
    Highlight your skills and experience, show your portfolio, and set your ideal pay rate.
  • Talent comes to you icon
    Work the way you want
    Apply for jobs, create easy-to-by projects, or access exclusive opportunities that come to you.
  • Payment simplified icon
    Get paid securely
    From contract to payment, we help you work safely and get paid securely.
Want to get started? Create a profile

About Upwork

  • Rating is 4.9 out of 5.
    4.9/5
    (Average rating of clients by professionals)
  • G2 2021
    #1 freelance platform
  • 49,000+
    Signed contract every week
  • $2.3B
    Freelancers earned on Upwork in 2020

Find the best freelance jobs

Growing your career is as easy as creating a free profile and finding work like this that fits your skills.

Trusted by

  • Microsoft Logo
  • Airbnb Logo
  • Bissell Logo
  • GoDaddy Logo