The "Best friend's sister" angle, huh? I've been down this road a few times, successfully. It's a very intriguing path to go down. Now, the main thing here to remember is that this is the sister of your best friend. Duh, sounds obvious, right? But many people forget that, and just flirt like it's any other girl. Nay nay, don't do this.
I mean, chances are, your best friend has heard many a story about how you flirt with girls, what you think about the hot ones, and what you may think negatively about an old fling or an ex. All of this will come to light in his mind once you legitimately pursue his sister, and he probably will be at least a little apprehensive about this. That said, that's the only real bad news about this. As long as you are respectful, your relationship with your friend should be fine.
Now, the sister is another story. She may be hung up on the days where you used to eat paste, and may never see you as more than, like you say, an annoying little brother.
But if you can make it clear to her that you're interested, she'll definitely consider it, and probably think about it a lot because it is quite the conundrum. Stay calm, act natural and get your point across; you'll win. DO'S for the party: Read her signs - if she's not feeling it right away, back off, try again later Show your more mature side DON'Ts for the party: Don't ingest too much liquid courage - sloppy is childish and you don't want her seeing you childish Grind overly sexual with her.
This will strain your relationship with besty. Try and take things to a bedroom. Just try to keep the mood light, man. You can do this, it's not uncommon.