Please do this. Try to keep to time. This is really important. It's being considerate to your colleagues on the panel, and also to the people who would like to hear papers on two panels. Your paper is probably interesting. It might be the most interesting paper in the world. Everybody might want to hear more, but you know? that's what question time is for. If it is the most interesting paper in the world, the questions will reflect that.
I have been at conferences where senior scholars have gone over -- a lot (one went over 40 minutes, and made it necessary to carry the questions over to the next morning, so that the two senior scholars who had stuck to time -- and incidentally given much more interesting and solid papers -- could answer questions). It's really unforgivable. But if you are a senior scholar, especially one who is well-known, you can get away with it once in a while. If you are a well-loved senior scholar, you might be able to get away with it more often, but if you make a habit of it, you won't be as well-loved.
If you are a junior scholar, it depends on how good you are. If you're scary good, then you can probably get away with it in the manner of senior scholars. But if you aren't? Best to be extra-polite.
If you are a grad student? We all get how involved you are with your subject. We were there once, too. But there's a good case that you aren't as plugged in to the community at the conference yet. You might want to consider that people on your panel and in the audience are folks who can be useful to you, or whom you want to impress, but you ave no idea what they look like. And this can be true on a larger level. When, for example, someone who appears meek and polite (not me, btw) makes a comment and asks a question, think carefully before correcting them abruptly. That person could be someone whose research and teaching have included your topic since before you started grad school, and the question might be going somewhere that would help you. They might be making a different point that you weren't expecting. And sometimes, that person is also a person who organizes conferences in your field, or edits a journal you want to submit to, or will recognize your paper when it comes across her desk for peer review.
I happen to be a person who teeters between absolute fear at conferences (yes, I spend hours asking patient friends if my paper was ok, or if my question was dumb!) and trying to be really polite, and then stepping into things and being perhaps too blunt (and in fact, I just jumped in and argued with a colleague over something). There are many people who are better at conference behavior than I am. But I do think that considering all the dynamics of what could be happening around you, and that every person you don't know (or do), might be worth at least trying to be polite to, is probably something to aim for.