It sounds like you should be using a dedicated video streaming provider for your videos. At the moment the user has to download them from your application – this is causing the delay.
If you do go for the streaming approach, once the video has been uploaded to your application, you’d just need a scheduled job to or queued job to upload it onto the streaming provider, and make a note of the playback URL. When a user comes to watch the video, just direct the browser to the stream rather than downloading the entire video.