We want to run custom lightning component in the lightbox for quick action. Is there any way to display the same layout but for custom component?
You'll want to read Configure Components for Custom Actions and its related documentation, but basically you will want to implement force:lightningQuickAction or force:lightningQuickActionWithoutHeader, and if you want object-specific actions, also include force:hasRecordId. Try this as an example:
<aura:component implements="force:lightningQuickAction,force:hasRecordId"> Hello World! </aura:component>
Once you create the component, go in to Setup, create a new quick action (e.g. Setup > Object Manager > Account > Buttons, Links, and Actions) using the component, then you can add it to your page layout. Once you've got the basic understanding of how it works, you'll be able to develop actions that can be used globally, per-object, or both, and choose to customize the buttons, etc. Everything you need is in the documentation. If you have specific questions, please feel free to follow up with a new question.